82
7/23/2019 Andrey Nikolaev Latch Internals 2012 for Ruoug http://slidepdf.com/reader/full/andrey-nikolaev-latch-internals-2012-for-ruoug 1/82 Latch internals RUOUG Seminar, December 6, 2012 Andrey Nikolaev Oracle Database Performance Expert RDTEX, Russia

Andrey Nikolaev Latch Internals 2012 for Ruoug

  • Upload
    errr33

  • View
    229

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 182

Latch internalsRUOUG Seminar December 6 2012

Andrey NikolaevOracle Database Performance ExpertRDTEX Russia

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 282

WHO AM I

bull AndreyNikolaevrdtexru

bull RDTEX First Line Support Centerbull httpandreynikolaevwordpresscom

bull Latch mutex and beyond

bull Specialize in Oracle performance tuningbull Over 20 years of Oracle related experience as a research

scientist developer DBA performance consultant lecturerhellip

bull Occasionally presented at conferencesbull RUOUG member

RDTEX | Latch internals | RUOUG Seminar

2

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 382

EPISODE OF LATCH CONTENTIONbull Oracle instance hangs due to heavy cache buffers chains latch contention

3

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 482

WHAT WE ALL KNEW ABOUT ORACLE LATCHES

bull Latches spin up to the instance parameter _SPIN_COUNT

that defaults to 2000bull This number can be tuned dynamically

bull Latch waits use increasing exponential backoff

bull Latch waits are very shortbull We should tune application not the _SPIN_COUNT

RDTEX | Latch internals | RUOUG Seminar

4

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 582

CONTEMPORARY LATCHES SINCE 92

RDTEX | Latch internals | RUOUG Seminar

5

I was wondering to find that

bull Exclusive latch spins 10 times morebull The exponential backoff was disappeared more then decade

ago in Oracle 92

bull All except one latch use wait posting now

bull Latch free wait may be infinite

bull Sometimes it is worth to tune the _SPIN_COUNT

bull This parameter is static for most latches

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 682

GREAT SOURCES OF LATCH INFO

ldquoOracle8i Internal Services for Waits Latches Locks and

Memoryrdquo by Steve Adams [1] 1999 ndash Founded new era of Oracle performance tuning

ndash However the Oracle latch technologies were evolved greatly duringthe decade

ldquoSystematic Latch Contention Troubleshootingrdquo approach byTanel Poder [2]

ndash Latchprofxsql script revolutionized the latch tuning

ldquoOracle Core Essential Internals for DBAs and Developersby Jonathan Lewis [3] 2011

RDTEX | Latch internals | RUOUG Seminar

6

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 782

REVIEW OF SERIALIZATION MECHANISMS IN ORACLE

bull Latches are simple low-level serialization mechanisms that coordinatemultiuser access to shared data structures objects and fileshellip

bull A mutual exclusion object (mutex) is a low-level mechanism thatprevents an object in memory from aging out or from being corrupted hellip

bull Internal locks are higher-level more complex mechanisms hellip

Oraclereg Database Concepts 112

bull KGX Mutexes appeared in latest Oracle versions inside Library Cache

RDTEX | Latch internals | RUOUG Seminar

A983139983139983141983155983155

A983139983153983157983145983155983145983156983145983151983150

983123983152983145983150983148983151983139983147 983138983137983155983141983140

983124983145983149983141983155983139983137983148983141

983116983145983142983141983139983161983139983148983141

983116983151983139983147983155

983123983141983158983141983154983137983148 983117983151983140983141983155

FIF983119

983118983151

gt 983117983145983148983148983145983155983141983139983151983150983140983155

D983161983150983137983149983145983139

983116983137983156983139983144983141983155

983124983161983152983141983155 983137983150983140 983117983151983140983141983155

983123I983122983119 (983155983152983145983150) + FIF983119

983129983141983155

983117983145983139983154983151983155983141983139983151983150983140983155

983123983156983137983156983145983139

983117983157983156983141983160983141983155

983119983152983141983154983137983156983145983151983150983155

983123I983122983119

983129983141983155

983123983157983138983117983145983139983154983151983155983141983139983151983150983140983155

D983161983150983137983149983145983139 7

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 882

HOW SPINLOCK WORKS

RDTEX | Latch internals | RUOUG Seminar

8

CPU3

Free FreeHolding by CPU 1

Spin polls the spinlock location

Get

Miss Sleep Spin count ( )∆

Holding time (S )

CPU2 Spin Holding the spinlock

Holding by CPU 2

CPU1 Holding the spinlock

The spinlock location

Sleep releases the CPU

bull Oracle latches

ndash Use atomic hardware instruction for Immediate get

ndash If missed process spins by polling latch location during Spin get

ndash Number of spin cycles is bounded by spin count

ndash In spin get not succeed the process acquiring the latch sleeps

hellip

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 982

CLASSIC SPINLOCKS

bull Wiki hellip spinlock hellip waits in a loop repeatedly checking until the lockbecomes available hellip

bull Introduced by Edsger Dijkstra [4] in 1965 Have been thoroughlyinvestigated since that time [5]

bull Many sophisticated spinlock realizations were proposed and evaluated

(TS TTS Delay MCS Anderson)

bull Two general types

ndash System spinlocks Kernel OS threads cannot wait Major metrics

atomic operations frequency Shared bus utilization

ndash User spinlocks Oracle latch and mutex Average lock holding time ~10 us It is more efficient to poll a lock rather than pre-empt the threaddoing 1 ms context switch Metrics CPU and elapsed times

RDTEX | Latch internals | RUOUG Seminar

9

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1082

THE NEED FOR SPIN

RDTEX | Latch internals | RUOUG Seminar

10

Time to complete CBC latch contention testcase vs number of threads

No spinning _spin_count=0

_spin_count=2000

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1182

SPINLOCK REALIZATIONS

RDTEX | Latch internals | RUOUG Seminar

Spinlock Pseudocode Problems

TS

pre-112 mutex

while(Test_and_Set(lock)) Bus saturation

TTS

Oracle latch

while(lock||Test_and_Set(lock)) Invalidation storms onrelease (ldquoopen doorrdquo

ldquothundering herdsrdquo)

Delay

Latch and 112

Mutex

Adjustable delay Slower under contention

Anderson MCSetc

Queues Widely used in JavaLinux kernel hellip

CPU and memoryoverhead preemptionissues

11

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1282

THE TOOLS

RDTEX | Latch internals | RUOUG Seminar12

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1382

DTRACE SOLARIS DYNAMIC TRACING FRAMEWORK

DTrace allows us to investigate how Oracle latches perform in real time ndash Create triggers on any event inside Solaris and function call inside Oracle

providermodulefunctionname

pid1910oraclekslgetlentry ndash Write trigger bodies ndash actions May read and change any address location ndash Can count the latch spins trace the latch waits perform experiments ndash Measure times and distributions up to microseconds precision

RDTEX | Latch internals | RUOUG Seminar

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313500

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1482

ORADEBUG INTERNAL ORACLE DEBUGGER

RDTEX | Latch internals | RUOUG Seminar

bull oradebug call Allows us invoke any internal Oracle function manually

SQLgt oradebug peek 200222A0 24

[200222A0 200222B8) = 00000016 00000001 000001D0 00000007

SQLgt oradebug call kslgetl 0x5001A020 1 2 3Function returned 1

bull oradebug peek Examines contents of any memory location

bull oradebug poke Modifies memory

bull oradebug watch Watch a region of memory

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1582

LATCH CONTENTION TESTCASES

RDTEX | Latch internals | RUOUG Seminar

15

create table latch_contention as select from dualhellipfor i in 11000000loopselect dummy into a from latch_contention

end loop

bull Exclusive and shared latches behave differentlybull Contention for shared cache buffer chains latch arises when manysessions scan the same block concurrently and _fastpin_enable=0

bull To induce the contention for exclusive ldquorow cache objectsrdquo latch protectingdc_users row cache just add several dynamic RLS policies

exec dbms_rlsadd_policy(systemlatch_contentionrls_1 hellip

exec dbms_rlsadd_policy(systemlatch_contentionrls_2 helliphellip

bull These dummy policies generate predicates 1=1bull Many other contention scenarios possible

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313509831351

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1682

ROW CACHE LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

16

Elapsed and CPU times vs number of threadsOracle Database Appliance (24 SMT)

Latch wait

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1782

HOW ORACLEREQUESTS THE LATCH

RDTEX | Latch internals | RUOUG Seminar17

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1882

DTRACE REVEALS LATCH INTERFACE ROUTINES

Oracle calls the following functions to acquire the latch

ndash kslgetl(laddr wait why where) - get exclusive latch ndash kslg2c(l1l2trcwhy where) - get two excl child latches

ndash kslgetsl (laddrwaitwhywheremode) - get shared latch

bull 11g - ksl_get_shared_latch (hellip)

ndash kslg2cs(l1l2modetrcwhy where)) - get two shr child latches

ndash kslgpl(laddrcommentwhywhere) - get parent and all childs

ndash kslfre(laddr) - free the latch

Oracle give us possibility to do the same by oradebug call

RDTEX | Latch internals | RUOUG Seminar

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313501

983125983155983141 32983085983138983145983156 983152983151983154983156983155

64983138983145983156 O983154983137983139983148983141 983139983151983154983154983157983152983156983155 983156983144983141 983137983154983143983157983149983141983150983156983155 983151983142 983151983154983137983140983141983138983157983143 983139983137983148983148 983145983150 10204 983137983150983140 983137983138983151983158983141

18

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1982

ldquoWHERErdquo AND ldquoWHYrdquo ORACLE GETS THE LATCH

To request the latch Oracle kernel routines need

bull laddres ndash address of latch in SGAbull wait ndash flag for no-wait (0) or wait (1) latch acquisition

bull where ndash integer code for location from where the latch is acquired

Oracle lists possible latch ldquowhererdquo values in x$ksllwindxbull why - context of why the latch is acquiring at this ldquowhererdquo It

contains DBA for CBC latches (Tanel Poder) SGA chunk addressfor shared pool latch session address etchellip ldquoWhyrdquo meaning for

some ldquowhererdquo can be found in x$ksllwksllwlbl

bull mode ndash requesting state for shared lathes

ndash 8 SHARED mode 16 EXCLUSIVE mode

RDTEX | Latch internals | RUOUG Seminar

19

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2082

LIST OF LATCH ldquoWHERErdquo LOCATIONS

X$KSLLW ndash List of latch [W]here Part of v$latch_misses

ndash ksllwnam ndash Kernel function name(ldquokghupr1rdquo ldquoksqdelrdquo etc) ndash ksllwlbl ndash Description of ldquowhyrdquo context

Join this code on indx to

ndash x$ksuprlatksulawhr (v$latchholder)ndash ldquowhererdquo value that wasused by latch holder

ndash x$kslltkslltwhr (v$latch) ndash ldquowhererdquo that was used by lastsuccessful latch acquisition

ndash x$ksuprksllawer (v$process) ndash ldquowhererdquo that was used byprocess is being waiting for latch

ndash x$kslwscindx (v$latch_misses) ndash latch sleep statistics by

ldquowhererdquo

RDTEX | Latch internals | RUOUG Seminar

20

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2182

V$LATCH_MISSES

bull Based on X$KSLWSC - No[W]ait and [S]leep [C]ount stats byContext Documentation is rather obscure

bull The following counters are incremented on each latch sleep

sleep_count (ldquowhererdquo the latch was held)

wtr_slp_count (ldquowhererdquo the latch get was requested)

bull nwfail_count is incremented only for session which SID is equalto _LATCH_MISS_STAT_SID parameter

bull longhold_count is incremented for shared latches only when

someone held the latch at this ldquowhererdquo for the entire duration ofsleep

RDTEX | Latch internals | RUOUG Seminar

21

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2282

LATCH IS HOLDING BY PROCESS NOT SESSION

bull

RDTEX | Latch internals | RUOUG Seminar

983123983156983154983157983139983156 983147983155983157983152983154 983163

991270

983123983156983154983157983139983156 983147983155983148983148983137983163983147983155983148983148983156 98314798315598314898314898313798314898313798315698313114983133

983165

991270983165

983120983154983151983139983141983155983155 983142983145983160983141983140 983137983154983154983137983161

983158$983152983154983151983139983141983155983155 983085gt 983160$983147983155983157983152983154

983123983156983154983157983139983156 983147983155983148983148983156983163

991270

983165

L983145983155983156 983151983142 983137983148983148 983148983137983156983139983144983141983155

983158$983148983137983156983139983144 983085gt983160$983147983155983148983148983156

Each process has an array of references to the latches it is holding

22

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2382

KSLLA IS HIDDEN IN V$PROCESS

bull The process latching info is the kslla structure embedded in theprocess state object

bull Some kslla fields were externalized to SQL

bull kslla contains array of latches holding by process at each level andinformation about the state of any current attempt to acquire a

latch Not externalized directly

bull When you query v$latchholder x$ksuprlat Oracle scansv$processx$ksupr in order to find all the processes that are

holding the latchesbull Tanel Poder uses high frequency sampling of v$latchholder to

systematically troubleshoot latch contention

RDTEX | Latch internals | RUOUG Seminar

23

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2482

X$KSUPRKSLLA FIELDS INSTRUMENT THE LATCH GET

bull ksllalaq ndash address of latch acquiring Populated during immediateget (and spin before 11g)

bull ksllawat - latch being waited for This is v$processlatchwait

bull ksllawhy ndash ldquowhyrdquo for the latch being waited for

bull ksllawere ndash ldquowhererdquo for the latch being waited for

bull ksllalow ndash bit array of levels of currently holding latches

bull ksllaspn - latch this process is spinning on v$processlatchspinNot populated since 81

bull ksllaps - inter-process post statistics

RDTEX | Latch internals | RUOUG Seminar

24

Generic ldquolatch freerdquo and 27 latch specific ldquolatch free rdquo wait events areassociated with sleep only Spin is invisible to Oracle Wait Interface

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2582

THE LATCH STRUCTURE ndash KSLLT

RDTEX | Latch internals | RUOUG Seminar

struct ksllt

ltLatchgt

ldquowhererdquo and ldquowhyrdquo

Level latch class other attributes

Statistics

Latch wait list header

hellip

25

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2682

LATCH SIZE BY VERSION

nix 32bit nix 64bit Windows 32bit

734 92 - 120

806 104 - 104817 104 144 104

901 200 160

920 196 240 200

1010 256 208

1020 - 11203 100 160 104

RDTEX | Latch internals | RUOUG Seminar

x$ksmfsv ndash list of all fixed SGA variables983123E983116EC983124 DI983123983124I983118C983124 983147983155983149983142983155983155983145983162

F983122983119983117 983160$983147983155983149983142983155983158

983127HE983122E 983147983155983149983142983155983156983161983152 = 983147983155983148983148983156

Latch structure was bigger in 101 due to additional latch statistics

26

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2782

ORACLE LATCH IS NOT JUST A SINGLE MEMORY LOCATION

bull Before 11g Value of first latch byte (word for shared latches) wasused to determine latch state

0x00 ndash latch is free

0xFF ndash exclusive latch is busy Was 0x01 in Oracle 7

0x010x02hellip - shared latch holding by 12 hellip processes simultaneously

0x20000000 | pid - shared latch holding exclusively

bull In 11g first latch word shows the pid of the latch holder

0x00 ndash latch is free0x12 ndash Oracle process with pid 18 holds the exclusive latch

RDTEX | Latch internals | RUOUG Seminar

27

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2882

BUSY ORACLE 11G LATCH IN MEMORY

bull 112 Exclusive latch (32-bit platform)

bull 112 Shared latch EXCLUSIVE mode

RDTEX | Latch internals | RUOUG Seminar

SQLgt oradebug peek 200222A0 24

[200222A0 200222B8) = 00000016 00000001 000001D0 00000007

pid^ gets latch level

SQLgt oradebug peek 0xampladdr 24

[2000C814 2000C82C) = 2000001F 0000007C 0000007E 00000006 pid^ gets latch level

983139983151983150983156 983155983153983148983152983148983157983155 983150983151983148983151983143 9831409831419831499831519831350128

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2982

LATCH FIXED TABLES

bull Before Oracle 11g all the latches were externalized to SQL inv$latch (x$ksllt)

bull In 11g parent (fixed SGA) latches are externalized inv$latch_parent (x$kslltr_parent)

bull In 11g child (allocated on startup) latches are externalized in

v$latch_children (x$kslltr_children) bull Oracle 11g v$latch (x$kslltr) merges these data Beware select

from x$kslltr acquires all parent latches and their children

bull Only part of latch structure is visible in x$ksllthellip fixed tables ndash kslltwhr kslltwhy ndash ldquowhererdquo and ldquowhyrdquo latch was acquired last time

ndash kslltwgt hellip - statistics

ndash kslltlvl hellip - attributes

ndash kslltwtt ndash latch wait time

RDTEX | Latch internals | RUOUG Seminar

29

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3082

CUMULATIVE V$LATCH STATISTIC COUNTERS

RDTEX | Latch internals | RUOUG Seminar

Statistics x$ksllt When and how it increments

GETS kslltwgt ldquo++rdquo after wait mode latch get

MISSES kslltwff ldquo++rdquo after wait get if it was missed

SLEEPS kslltwsl ldquo+number_of_sleepsrdquo during get

SPIN_GETS ksllthst0 ldquo++rdquo if get was missed but not slept

WAIT_TIME kslltwtt ldquo+wait_timerdquo after latch get

IMMEDIATE_GETS kslltngt ldquo++rdquo after nowait mode latch get Is notincremented on miss

IMMEDIATE_MISSES kslltnfa ldquo++rdquo if nowait mode get was missed

30

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3182

MANY STATISTICS WERE LOST IN 102

In 102 Oracle made obsolete many useful latch statistics

Latch size was decreased and the following fields always contain 0now in v$latch

ndash v$latchwaiters_woken

ndash v$latchwaits_holding_latch

ndash v$latchsleep1hellipsleep10

ndash and many undocumented statistics

This was mentioned in 102 Reference

RDTEX | Latch internals | RUOUG Seminar

31

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3282

X$KSLLD - LATCH DESCRIPTORS

bull Flags and attributes describing each latch are stored in kslldt arrayof kslld structures

bull [K]ernel [S]ervice [L]atch [L]ock [D]escriptor

bull Some but not all of kslld fields are externalized via x$kslld andv$latchname

bull Located in the PGA

RDTEX | Latch internals | RUOUG Seminar

32

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382

LATCH ATTRIBUTES

bull Each latch have at least the following attributes in kslldt

ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared

ndash PAR Is the latch Solitary or Parent for the family of child latches

ndash G2C Can two child latches be simultaneously requested in wait mode

using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92

ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above

ndash Level 0-14 Used to prevent latch deadlocks

ndash Class 0-7 Spin and wait class assigned to the latch 92 and above

RDTEX | Latch internals | RUOUG Seminar

33

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482

LATCH ATTRIBUTES BY ORACLE VERSION

RDTEX | Latch internals | RUOUG Seminar

11203 553 154 72 - 6 93

Oracle Number of latches PAR G2C LNG UFS SHARED

7340 53 14 2 3 - -

8063 80 21 7 3 - 38174 152 48 19 4 - 9

9208 242 79 37 - - 19

10202 385 114 55 - 4 47

10203 388 117 58 - 4 48

10204 394 117 59 - 4 50

11106 496 145 67 - 6 81

11107 502 145 67 - 6 83

11201 535 149 70 - 6 86

11202 551 153 72 - 6 91

34

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582

LATCH LEVELS

bull Every latch has a level in the range 0-14 to avoid deadlocks

bull Process can request a latch X in wait mode after obtaining latch Y if and

only if level X gt level Y

bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y

bull If these rules are broken the Oracle process raises

ORA-600[504] - trying to obtain another latch at incompatible level Note 281041

ORA-600[525] - trying to obtain another latch at the same level Note 1388881

ORA-600[526] - latch child order violation Note 1388891

RDTEX | Latch internals | RUOUG Seminar

35

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682

PARENT AND CHILD LATCHES

bull Latch can be a parent child or solitary latch

bull Both parent and child latches share the same latch name

bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss

bull Child latches are allocated during instance startup

bull The parent latch can be gotten independently

bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s

RDTEX | Latch internals | RUOUG Seminar

(latch info) wait_event=0 bits=10

holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl

Context saved from call 0

state=busy

36

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782

LATCH TREES

ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches

Direct SGA access program output for 9206 instance with too smallshared pool

RDTEX | Latch internals | RUOUG Seminar

ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1

waiter ospid 129 sid 72 pid 45

holding a154b7120 library cache (157) level=5 child=17 kglupc child

waiter ospid 18255 sid 65 pid 930

waiter ospid 6690 sid 554 pid 1654

waiter ospid 4685 sid 879 pid 1034

hellip

waiter ospid 29749 sid 180 pid 155

holding a154b7db8 library cache (157) level=5 child=4 kglupc child

waiter ospid 13104 sid 281 pid 220

waiter ospid 24089 sid 565 pid 636

waiter ospid 25002 sid 621 pid 1481

waiter ospid 16930 sid 1046 pid 783hellip

37

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882

LATCH CLASSES

bull Classes with different spin and wait policies

bull Up to 8 classes By default all the latches belong to class 0

bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2

bull Latch can be assigned to class by its number For example

ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000

ndash _LATCH_CLASSES=46

bull Will be discussed later

RDTEX | Latch internals | RUOUG Seminar

SQLgt select from x$ksllclass

INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4

---- ------ ----- -------- ------ ------ ------ ------ ------

0 20000 0 1 8000 8000 8000 8000 8000

hellip

6 100 12 1000 2000 3000 4000 4000 4000

38

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982

LATCH ACQUISITION IN NO-WAIT MODE

bull kslgetl(laddr0whywhere) ndash Exclusive latch get

bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep

ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches

ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches

bull IMMEDIATE_ statistics are updated

bull Used when many candidate latches are available to bypass the

latch level rule or as part of CBC chain examination

RDTEX | Latch internals | RUOUG Seminar

sqlplus nolog demo_03sql39

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082

LATCH WAITS

RDTEX | Latch internals | RUOUG Seminar 40

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182

WAITING FOR THE LATCH

RDTEX | Latch internals | RUOUG Seminar

983123 G A

C983120983125 1 C983120983125 2

983120983154983151983139983141983155983155 A

983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)

983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144

983116983137983156983139983144

983120983154983151983139983141983155983155 B

41

RDTEX | L h i l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282

LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION

bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries

again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of

times

bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep

bull A sleep corresponds to a wait for the latch free wait event

bull Initially a process sleeps for 001 s This time is doubled in every

subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip

RDTEX | Latch internals | RUOUG Seminar

Still did not found any versionwhich used this

May be it was lt= Oracle 6

42

RDTEX | L t h i t l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382

LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81

Latch gets with wait (kslgetl(laddress1hellip))

ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times

ndash Sleep on latch free event with exponential backoff

ndash Repeat

RDTEX | Latch internals | RUOUG Seminar

43

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482

8I LATCH SPIN CODE FLOW USING DTRACE

RDTEX | Latch internals | RUOUG Seminar

bull hellip Event 10046 trace

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2

983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29

983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270

sqlplus nolog demo_04_8sql 29

44

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582

POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF

bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec

RDTEX | Latch internals | RUOUG Seminar

12 ]2 )1[(

minus= +wait N

timeout

45

bull Most waits were for nothing ndash latch already was free

bull Latch utilization can not be high

bull Unnecessary spins provoke CPU thrashing

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682

92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE

sqlplus nolog demo_05sql 45

RDTEX | Latch internals | RUOUG Seminar

kslgetl(0x50006318 1)

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt kslges(0x50006318 ) -wait latch get

-gt skgslsgts(0x50006318 ) -spin latch get

-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT

-gt kskthbwt(0x0)

-gt kslwlmod() - set up Wait List

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt skgpwwait -sleep latch getsemop(11 17-10 1)

983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140

46

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782

RELIABLE LATCH WAITS

bull Hidden latch wait revolution that we missed Latch relies on waitposting

bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout

bull Exclusive latch spins 20000 cycles by default

bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock

bull Latches assigned to non-default class wait until timeout

bull Version 90 was transient and used mix of backoff and wait posting

This is why _LATCH_WAIT_POSTING parameter disappeared in9i

bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin

RDTEX | Latch internals | RUOUG Seminar

47

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882

IMAGINE THE TIME MICROSCOPE

| a a s | a

Reality One million zoomed time

1 us 1 sec

1 ms 17 min1 sec 115 days

Light speed (300000 kmsec) Sonic speed (300 msec)

CPU tick (2 GHz) 00005 sec

Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec

Max spin time for shared latch (~220005 ticks) ~10 sec

Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec

Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec

OS context switch time (10us-1ms) 10sec-17min

48

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982

LATCH WAITS UNDER TIME MICROSCOPE

Old 8i latch algorithmSpin for latch during 5 seconds

Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min

hellipSpin again for 5 seconds

Sleep again for 23 days (2 sec) hellip

New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)

If spin not succeeded go to sleep until post

According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)

| |

49

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082

_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES

bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and

_LATCH_CLASS_0 parameters

bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000

bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter

bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch

50

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182

TIMED LATCH WAIT POSTING

bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel

bull This can lead to instance hang ndash process will never be woken up

bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout

bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem

bull Generic AIX ports always use timed latch wait posting

bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux

51

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282

FINE GRAIN LATCH TUNING

bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting

bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7

bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo

microseconds using pollsys() (not semtimedop()) system call

ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times

Loop up to ldquoSpinsrdquo cycles

Yield CPU using yield() (or sched_yield())

ndash Sleep for ldquoSleepXrdquo usecs

ndash Then spin again hellip

52

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382

TRACES OF NONDEFAULT CLASS LATCH

983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152

991270

9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000

991270 E983158983141983150983156 10046 983156983154983137983139983141

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482

PERFORMANCE OF LATCH CLASSES

Row cache objects latch contention testcase on ODA (24 SMT)54

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582

IS THE _SPIN_COUNT REALLY STATIC

bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch

waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used

ldquocache buffers chainrdquo latch contention

bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass

(_LATCH_CLASS_X) By default ndash 20000

bull Shared latch spin can be tuned by _SPIN_COUNT value By

default ndash 2000 However this is not trivial

55

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682

SHARED LATCH BEHAVE LIKE ENQUEUE

bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80

bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests

bull This blocking achieved by special 0x40000000 bit in the latch

value This bit is a flag that some incompatible latch acquisitionis in progress

bull Queued processes will be woken up one by one on latch release

bull X mode latch gets effectively serialize the shared latch

bull See my blog for description of experiments

56

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782

TRACE OF SHARED LATCH WAIT

Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the

latch wait list and then spins again

983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141

983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156

983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983147983155983148983159983148983149983151983140()

983147983155983148983155983147983143983155(098316050009BAC )

983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140

983155983141983149983151983152(27)

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882

SHARED LATCH ACQUISITION

bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned

bull X mode shared latch get spins by default up to 4000 cycles

bull S mode does not spin at all (or spins in unknown way)

S mode get X mode get

Held in S mode Compatible 2_SPIN_COUNT

Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT

58

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982

LATCH RELEASE

bull Free the latch ndash kslfre(laddr)

bull Oracle process releases the latch nonatomically

bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process

bull This requires less bus invalidation and ensures propagation of latch

release to other local cachesbull Not fair policy Spinners on the local CPU board have the

preference

bull Then process posts the first process in the list of waiters

59

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082

LATCH CONTENTION

RDTEX | Latch internals | RUOUG Seminar60

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182

LATCH CONTENTION

bull Occurs when a latch is required by several processes at thesame time

bull Mostly due to bugs or abnormal application behavior

bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp

bull To diagnose the latch contention we need to investigate thelatch statistics [8]

61

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282

DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS

Latch requests arrival rate

Immediate gets efficiency

Latch sleeps ratio

Latch wait time per second

Latch spin efficiencymisses

getsspin∆

∆= _σ

misses

sleepsK

∆=

time

gets

∆=λ

gets

misses

∆= ρ

time

timewait W

∆=

_

Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics

62

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382

SAMPLED AVERAGES

By sampling of x$ksupr we can estimate

ndash Length of latch wait queue

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS

ndash Number of spinning processes in 10g

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS

Sampling of v$lathholder estimates latch utilization

Using DTrace we can measure

ndash Latch acquisition and holding times

ndash Actual spin count

w L

s N

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482

DERIVED LATCH STATISTICS

Latch utilization (PASTA)

Average holding time

Length of latch wait list

Recurrent sleeps ratio

Latch acquisition time

time

timeholdinglatchU

∆=asymp

__ ρ

Re_100

___

questsGet

TimeSnap MissGet Pct S

lowast==

λ

W L =

κ

κ σ 1minus+

)(1

W N T saq += minusλ

64

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 2: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 282

WHO AM I

bull AndreyNikolaevrdtexru

bull RDTEX First Line Support Centerbull httpandreynikolaevwordpresscom

bull Latch mutex and beyond

bull Specialize in Oracle performance tuningbull Over 20 years of Oracle related experience as a research

scientist developer DBA performance consultant lecturerhellip

bull Occasionally presented at conferencesbull RUOUG member

RDTEX | Latch internals | RUOUG Seminar

2

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 382

EPISODE OF LATCH CONTENTIONbull Oracle instance hangs due to heavy cache buffers chains latch contention

3

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 482

WHAT WE ALL KNEW ABOUT ORACLE LATCHES

bull Latches spin up to the instance parameter _SPIN_COUNT

that defaults to 2000bull This number can be tuned dynamically

bull Latch waits use increasing exponential backoff

bull Latch waits are very shortbull We should tune application not the _SPIN_COUNT

RDTEX | Latch internals | RUOUG Seminar

4

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 582

CONTEMPORARY LATCHES SINCE 92

RDTEX | Latch internals | RUOUG Seminar

5

I was wondering to find that

bull Exclusive latch spins 10 times morebull The exponential backoff was disappeared more then decade

ago in Oracle 92

bull All except one latch use wait posting now

bull Latch free wait may be infinite

bull Sometimes it is worth to tune the _SPIN_COUNT

bull This parameter is static for most latches

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 682

GREAT SOURCES OF LATCH INFO

ldquoOracle8i Internal Services for Waits Latches Locks and

Memoryrdquo by Steve Adams [1] 1999 ndash Founded new era of Oracle performance tuning

ndash However the Oracle latch technologies were evolved greatly duringthe decade

ldquoSystematic Latch Contention Troubleshootingrdquo approach byTanel Poder [2]

ndash Latchprofxsql script revolutionized the latch tuning

ldquoOracle Core Essential Internals for DBAs and Developersby Jonathan Lewis [3] 2011

RDTEX | Latch internals | RUOUG Seminar

6

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 782

REVIEW OF SERIALIZATION MECHANISMS IN ORACLE

bull Latches are simple low-level serialization mechanisms that coordinatemultiuser access to shared data structures objects and fileshellip

bull A mutual exclusion object (mutex) is a low-level mechanism thatprevents an object in memory from aging out or from being corrupted hellip

bull Internal locks are higher-level more complex mechanisms hellip

Oraclereg Database Concepts 112

bull KGX Mutexes appeared in latest Oracle versions inside Library Cache

RDTEX | Latch internals | RUOUG Seminar

A983139983139983141983155983155

A983139983153983157983145983155983145983156983145983151983150

983123983152983145983150983148983151983139983147 983138983137983155983141983140

983124983145983149983141983155983139983137983148983141

983116983145983142983141983139983161983139983148983141

983116983151983139983147983155

983123983141983158983141983154983137983148 983117983151983140983141983155

FIF983119

983118983151

gt 983117983145983148983148983145983155983141983139983151983150983140983155

D983161983150983137983149983145983139

983116983137983156983139983144983141983155

983124983161983152983141983155 983137983150983140 983117983151983140983141983155

983123I983122983119 (983155983152983145983150) + FIF983119

983129983141983155

983117983145983139983154983151983155983141983139983151983150983140983155

983123983156983137983156983145983139

983117983157983156983141983160983141983155

983119983152983141983154983137983156983145983151983150983155

983123I983122983119

983129983141983155

983123983157983138983117983145983139983154983151983155983141983139983151983150983140983155

D983161983150983137983149983145983139 7

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 882

HOW SPINLOCK WORKS

RDTEX | Latch internals | RUOUG Seminar

8

CPU3

Free FreeHolding by CPU 1

Spin polls the spinlock location

Get

Miss Sleep Spin count ( )∆

Holding time (S )

CPU2 Spin Holding the spinlock

Holding by CPU 2

CPU1 Holding the spinlock

The spinlock location

Sleep releases the CPU

bull Oracle latches

ndash Use atomic hardware instruction for Immediate get

ndash If missed process spins by polling latch location during Spin get

ndash Number of spin cycles is bounded by spin count

ndash In spin get not succeed the process acquiring the latch sleeps

hellip

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 982

CLASSIC SPINLOCKS

bull Wiki hellip spinlock hellip waits in a loop repeatedly checking until the lockbecomes available hellip

bull Introduced by Edsger Dijkstra [4] in 1965 Have been thoroughlyinvestigated since that time [5]

bull Many sophisticated spinlock realizations were proposed and evaluated

(TS TTS Delay MCS Anderson)

bull Two general types

ndash System spinlocks Kernel OS threads cannot wait Major metrics

atomic operations frequency Shared bus utilization

ndash User spinlocks Oracle latch and mutex Average lock holding time ~10 us It is more efficient to poll a lock rather than pre-empt the threaddoing 1 ms context switch Metrics CPU and elapsed times

RDTEX | Latch internals | RUOUG Seminar

9

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1082

THE NEED FOR SPIN

RDTEX | Latch internals | RUOUG Seminar

10

Time to complete CBC latch contention testcase vs number of threads

No spinning _spin_count=0

_spin_count=2000

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1182

SPINLOCK REALIZATIONS

RDTEX | Latch internals | RUOUG Seminar

Spinlock Pseudocode Problems

TS

pre-112 mutex

while(Test_and_Set(lock)) Bus saturation

TTS

Oracle latch

while(lock||Test_and_Set(lock)) Invalidation storms onrelease (ldquoopen doorrdquo

ldquothundering herdsrdquo)

Delay

Latch and 112

Mutex

Adjustable delay Slower under contention

Anderson MCSetc

Queues Widely used in JavaLinux kernel hellip

CPU and memoryoverhead preemptionissues

11

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1282

THE TOOLS

RDTEX | Latch internals | RUOUG Seminar12

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1382

DTRACE SOLARIS DYNAMIC TRACING FRAMEWORK

DTrace allows us to investigate how Oracle latches perform in real time ndash Create triggers on any event inside Solaris and function call inside Oracle

providermodulefunctionname

pid1910oraclekslgetlentry ndash Write trigger bodies ndash actions May read and change any address location ndash Can count the latch spins trace the latch waits perform experiments ndash Measure times and distributions up to microseconds precision

RDTEX | Latch internals | RUOUG Seminar

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313500

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1482

ORADEBUG INTERNAL ORACLE DEBUGGER

RDTEX | Latch internals | RUOUG Seminar

bull oradebug call Allows us invoke any internal Oracle function manually

SQLgt oradebug peek 200222A0 24

[200222A0 200222B8) = 00000016 00000001 000001D0 00000007

SQLgt oradebug call kslgetl 0x5001A020 1 2 3Function returned 1

bull oradebug peek Examines contents of any memory location

bull oradebug poke Modifies memory

bull oradebug watch Watch a region of memory

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1582

LATCH CONTENTION TESTCASES

RDTEX | Latch internals | RUOUG Seminar

15

create table latch_contention as select from dualhellipfor i in 11000000loopselect dummy into a from latch_contention

end loop

bull Exclusive and shared latches behave differentlybull Contention for shared cache buffer chains latch arises when manysessions scan the same block concurrently and _fastpin_enable=0

bull To induce the contention for exclusive ldquorow cache objectsrdquo latch protectingdc_users row cache just add several dynamic RLS policies

exec dbms_rlsadd_policy(systemlatch_contentionrls_1 hellip

exec dbms_rlsadd_policy(systemlatch_contentionrls_2 helliphellip

bull These dummy policies generate predicates 1=1bull Many other contention scenarios possible

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313509831351

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1682

ROW CACHE LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

16

Elapsed and CPU times vs number of threadsOracle Database Appliance (24 SMT)

Latch wait

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1782

HOW ORACLEREQUESTS THE LATCH

RDTEX | Latch internals | RUOUG Seminar17

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1882

DTRACE REVEALS LATCH INTERFACE ROUTINES

Oracle calls the following functions to acquire the latch

ndash kslgetl(laddr wait why where) - get exclusive latch ndash kslg2c(l1l2trcwhy where) - get two excl child latches

ndash kslgetsl (laddrwaitwhywheremode) - get shared latch

bull 11g - ksl_get_shared_latch (hellip)

ndash kslg2cs(l1l2modetrcwhy where)) - get two shr child latches

ndash kslgpl(laddrcommentwhywhere) - get parent and all childs

ndash kslfre(laddr) - free the latch

Oracle give us possibility to do the same by oradebug call

RDTEX | Latch internals | RUOUG Seminar

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313501

983125983155983141 32983085983138983145983156 983152983151983154983156983155

64983138983145983156 O983154983137983139983148983141 983139983151983154983154983157983152983156983155 983156983144983141 983137983154983143983157983149983141983150983156983155 983151983142 983151983154983137983140983141983138983157983143 983139983137983148983148 983145983150 10204 983137983150983140 983137983138983151983158983141

18

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1982

ldquoWHERErdquo AND ldquoWHYrdquo ORACLE GETS THE LATCH

To request the latch Oracle kernel routines need

bull laddres ndash address of latch in SGAbull wait ndash flag for no-wait (0) or wait (1) latch acquisition

bull where ndash integer code for location from where the latch is acquired

Oracle lists possible latch ldquowhererdquo values in x$ksllwindxbull why - context of why the latch is acquiring at this ldquowhererdquo It

contains DBA for CBC latches (Tanel Poder) SGA chunk addressfor shared pool latch session address etchellip ldquoWhyrdquo meaning for

some ldquowhererdquo can be found in x$ksllwksllwlbl

bull mode ndash requesting state for shared lathes

ndash 8 SHARED mode 16 EXCLUSIVE mode

RDTEX | Latch internals | RUOUG Seminar

19

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2082

LIST OF LATCH ldquoWHERErdquo LOCATIONS

X$KSLLW ndash List of latch [W]here Part of v$latch_misses

ndash ksllwnam ndash Kernel function name(ldquokghupr1rdquo ldquoksqdelrdquo etc) ndash ksllwlbl ndash Description of ldquowhyrdquo context

Join this code on indx to

ndash x$ksuprlatksulawhr (v$latchholder)ndash ldquowhererdquo value that wasused by latch holder

ndash x$kslltkslltwhr (v$latch) ndash ldquowhererdquo that was used by lastsuccessful latch acquisition

ndash x$ksuprksllawer (v$process) ndash ldquowhererdquo that was used byprocess is being waiting for latch

ndash x$kslwscindx (v$latch_misses) ndash latch sleep statistics by

ldquowhererdquo

RDTEX | Latch internals | RUOUG Seminar

20

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2182

V$LATCH_MISSES

bull Based on X$KSLWSC - No[W]ait and [S]leep [C]ount stats byContext Documentation is rather obscure

bull The following counters are incremented on each latch sleep

sleep_count (ldquowhererdquo the latch was held)

wtr_slp_count (ldquowhererdquo the latch get was requested)

bull nwfail_count is incremented only for session which SID is equalto _LATCH_MISS_STAT_SID parameter

bull longhold_count is incremented for shared latches only when

someone held the latch at this ldquowhererdquo for the entire duration ofsleep

RDTEX | Latch internals | RUOUG Seminar

21

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2282

LATCH IS HOLDING BY PROCESS NOT SESSION

bull

RDTEX | Latch internals | RUOUG Seminar

983123983156983154983157983139983156 983147983155983157983152983154 983163

991270

983123983156983154983157983139983156 983147983155983148983148983137983163983147983155983148983148983156 98314798315598314898314898313798314898313798315698313114983133

983165

991270983165

983120983154983151983139983141983155983155 983142983145983160983141983140 983137983154983154983137983161

983158$983152983154983151983139983141983155983155 983085gt 983160$983147983155983157983152983154

983123983156983154983157983139983156 983147983155983148983148983156983163

991270

983165

L983145983155983156 983151983142 983137983148983148 983148983137983156983139983144983141983155

983158$983148983137983156983139983144 983085gt983160$983147983155983148983148983156

Each process has an array of references to the latches it is holding

22

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2382

KSLLA IS HIDDEN IN V$PROCESS

bull The process latching info is the kslla structure embedded in theprocess state object

bull Some kslla fields were externalized to SQL

bull kslla contains array of latches holding by process at each level andinformation about the state of any current attempt to acquire a

latch Not externalized directly

bull When you query v$latchholder x$ksuprlat Oracle scansv$processx$ksupr in order to find all the processes that are

holding the latchesbull Tanel Poder uses high frequency sampling of v$latchholder to

systematically troubleshoot latch contention

RDTEX | Latch internals | RUOUG Seminar

23

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2482

X$KSUPRKSLLA FIELDS INSTRUMENT THE LATCH GET

bull ksllalaq ndash address of latch acquiring Populated during immediateget (and spin before 11g)

bull ksllawat - latch being waited for This is v$processlatchwait

bull ksllawhy ndash ldquowhyrdquo for the latch being waited for

bull ksllawere ndash ldquowhererdquo for the latch being waited for

bull ksllalow ndash bit array of levels of currently holding latches

bull ksllaspn - latch this process is spinning on v$processlatchspinNot populated since 81

bull ksllaps - inter-process post statistics

RDTEX | Latch internals | RUOUG Seminar

24

Generic ldquolatch freerdquo and 27 latch specific ldquolatch free rdquo wait events areassociated with sleep only Spin is invisible to Oracle Wait Interface

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2582

THE LATCH STRUCTURE ndash KSLLT

RDTEX | Latch internals | RUOUG Seminar

struct ksllt

ltLatchgt

ldquowhererdquo and ldquowhyrdquo

Level latch class other attributes

Statistics

Latch wait list header

hellip

25

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2682

LATCH SIZE BY VERSION

nix 32bit nix 64bit Windows 32bit

734 92 - 120

806 104 - 104817 104 144 104

901 200 160

920 196 240 200

1010 256 208

1020 - 11203 100 160 104

RDTEX | Latch internals | RUOUG Seminar

x$ksmfsv ndash list of all fixed SGA variables983123E983116EC983124 DI983123983124I983118C983124 983147983155983149983142983155983155983145983162

F983122983119983117 983160$983147983155983149983142983155983158

983127HE983122E 983147983155983149983142983155983156983161983152 = 983147983155983148983148983156

Latch structure was bigger in 101 due to additional latch statistics

26

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2782

ORACLE LATCH IS NOT JUST A SINGLE MEMORY LOCATION

bull Before 11g Value of first latch byte (word for shared latches) wasused to determine latch state

0x00 ndash latch is free

0xFF ndash exclusive latch is busy Was 0x01 in Oracle 7

0x010x02hellip - shared latch holding by 12 hellip processes simultaneously

0x20000000 | pid - shared latch holding exclusively

bull In 11g first latch word shows the pid of the latch holder

0x00 ndash latch is free0x12 ndash Oracle process with pid 18 holds the exclusive latch

RDTEX | Latch internals | RUOUG Seminar

27

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2882

BUSY ORACLE 11G LATCH IN MEMORY

bull 112 Exclusive latch (32-bit platform)

bull 112 Shared latch EXCLUSIVE mode

RDTEX | Latch internals | RUOUG Seminar

SQLgt oradebug peek 200222A0 24

[200222A0 200222B8) = 00000016 00000001 000001D0 00000007

pid^ gets latch level

SQLgt oradebug peek 0xampladdr 24

[2000C814 2000C82C) = 2000001F 0000007C 0000007E 00000006 pid^ gets latch level

983139983151983150983156 983155983153983148983152983148983157983155 983150983151983148983151983143 9831409831419831499831519831350128

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2982

LATCH FIXED TABLES

bull Before Oracle 11g all the latches were externalized to SQL inv$latch (x$ksllt)

bull In 11g parent (fixed SGA) latches are externalized inv$latch_parent (x$kslltr_parent)

bull In 11g child (allocated on startup) latches are externalized in

v$latch_children (x$kslltr_children) bull Oracle 11g v$latch (x$kslltr) merges these data Beware select

from x$kslltr acquires all parent latches and their children

bull Only part of latch structure is visible in x$ksllthellip fixed tables ndash kslltwhr kslltwhy ndash ldquowhererdquo and ldquowhyrdquo latch was acquired last time

ndash kslltwgt hellip - statistics

ndash kslltlvl hellip - attributes

ndash kslltwtt ndash latch wait time

RDTEX | Latch internals | RUOUG Seminar

29

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3082

CUMULATIVE V$LATCH STATISTIC COUNTERS

RDTEX | Latch internals | RUOUG Seminar

Statistics x$ksllt When and how it increments

GETS kslltwgt ldquo++rdquo after wait mode latch get

MISSES kslltwff ldquo++rdquo after wait get if it was missed

SLEEPS kslltwsl ldquo+number_of_sleepsrdquo during get

SPIN_GETS ksllthst0 ldquo++rdquo if get was missed but not slept

WAIT_TIME kslltwtt ldquo+wait_timerdquo after latch get

IMMEDIATE_GETS kslltngt ldquo++rdquo after nowait mode latch get Is notincremented on miss

IMMEDIATE_MISSES kslltnfa ldquo++rdquo if nowait mode get was missed

30

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3182

MANY STATISTICS WERE LOST IN 102

In 102 Oracle made obsolete many useful latch statistics

Latch size was decreased and the following fields always contain 0now in v$latch

ndash v$latchwaiters_woken

ndash v$latchwaits_holding_latch

ndash v$latchsleep1hellipsleep10

ndash and many undocumented statistics

This was mentioned in 102 Reference

RDTEX | Latch internals | RUOUG Seminar

31

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3282

X$KSLLD - LATCH DESCRIPTORS

bull Flags and attributes describing each latch are stored in kslldt arrayof kslld structures

bull [K]ernel [S]ervice [L]atch [L]ock [D]escriptor

bull Some but not all of kslld fields are externalized via x$kslld andv$latchname

bull Located in the PGA

RDTEX | Latch internals | RUOUG Seminar

32

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382

LATCH ATTRIBUTES

bull Each latch have at least the following attributes in kslldt

ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared

ndash PAR Is the latch Solitary or Parent for the family of child latches

ndash G2C Can two child latches be simultaneously requested in wait mode

using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92

ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above

ndash Level 0-14 Used to prevent latch deadlocks

ndash Class 0-7 Spin and wait class assigned to the latch 92 and above

RDTEX | Latch internals | RUOUG Seminar

33

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482

LATCH ATTRIBUTES BY ORACLE VERSION

RDTEX | Latch internals | RUOUG Seminar

11203 553 154 72 - 6 93

Oracle Number of latches PAR G2C LNG UFS SHARED

7340 53 14 2 3 - -

8063 80 21 7 3 - 38174 152 48 19 4 - 9

9208 242 79 37 - - 19

10202 385 114 55 - 4 47

10203 388 117 58 - 4 48

10204 394 117 59 - 4 50

11106 496 145 67 - 6 81

11107 502 145 67 - 6 83

11201 535 149 70 - 6 86

11202 551 153 72 - 6 91

34

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582

LATCH LEVELS

bull Every latch has a level in the range 0-14 to avoid deadlocks

bull Process can request a latch X in wait mode after obtaining latch Y if and

only if level X gt level Y

bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y

bull If these rules are broken the Oracle process raises

ORA-600[504] - trying to obtain another latch at incompatible level Note 281041

ORA-600[525] - trying to obtain another latch at the same level Note 1388881

ORA-600[526] - latch child order violation Note 1388891

RDTEX | Latch internals | RUOUG Seminar

35

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682

PARENT AND CHILD LATCHES

bull Latch can be a parent child or solitary latch

bull Both parent and child latches share the same latch name

bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss

bull Child latches are allocated during instance startup

bull The parent latch can be gotten independently

bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s

RDTEX | Latch internals | RUOUG Seminar

(latch info) wait_event=0 bits=10

holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl

Context saved from call 0

state=busy

36

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782

LATCH TREES

ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches

Direct SGA access program output for 9206 instance with too smallshared pool

RDTEX | Latch internals | RUOUG Seminar

ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1

waiter ospid 129 sid 72 pid 45

holding a154b7120 library cache (157) level=5 child=17 kglupc child

waiter ospid 18255 sid 65 pid 930

waiter ospid 6690 sid 554 pid 1654

waiter ospid 4685 sid 879 pid 1034

hellip

waiter ospid 29749 sid 180 pid 155

holding a154b7db8 library cache (157) level=5 child=4 kglupc child

waiter ospid 13104 sid 281 pid 220

waiter ospid 24089 sid 565 pid 636

waiter ospid 25002 sid 621 pid 1481

waiter ospid 16930 sid 1046 pid 783hellip

37

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882

LATCH CLASSES

bull Classes with different spin and wait policies

bull Up to 8 classes By default all the latches belong to class 0

bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2

bull Latch can be assigned to class by its number For example

ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000

ndash _LATCH_CLASSES=46

bull Will be discussed later

RDTEX | Latch internals | RUOUG Seminar

SQLgt select from x$ksllclass

INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4

---- ------ ----- -------- ------ ------ ------ ------ ------

0 20000 0 1 8000 8000 8000 8000 8000

hellip

6 100 12 1000 2000 3000 4000 4000 4000

38

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982

LATCH ACQUISITION IN NO-WAIT MODE

bull kslgetl(laddr0whywhere) ndash Exclusive latch get

bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep

ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches

ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches

bull IMMEDIATE_ statistics are updated

bull Used when many candidate latches are available to bypass the

latch level rule or as part of CBC chain examination

RDTEX | Latch internals | RUOUG Seminar

sqlplus nolog demo_03sql39

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082

LATCH WAITS

RDTEX | Latch internals | RUOUG Seminar 40

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182

WAITING FOR THE LATCH

RDTEX | Latch internals | RUOUG Seminar

983123 G A

C983120983125 1 C983120983125 2

983120983154983151983139983141983155983155 A

983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)

983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144

983116983137983156983139983144

983120983154983151983139983141983155983155 B

41

RDTEX | L h i l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282

LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION

bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries

again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of

times

bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep

bull A sleep corresponds to a wait for the latch free wait event

bull Initially a process sleeps for 001 s This time is doubled in every

subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip

RDTEX | Latch internals | RUOUG Seminar

Still did not found any versionwhich used this

May be it was lt= Oracle 6

42

RDTEX | L t h i t l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382

LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81

Latch gets with wait (kslgetl(laddress1hellip))

ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times

ndash Sleep on latch free event with exponential backoff

ndash Repeat

RDTEX | Latch internals | RUOUG Seminar

43

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482

8I LATCH SPIN CODE FLOW USING DTRACE

RDTEX | Latch internals | RUOUG Seminar

bull hellip Event 10046 trace

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2

983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29

983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270

sqlplus nolog demo_04_8sql 29

44

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582

POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF

bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec

RDTEX | Latch internals | RUOUG Seminar

12 ]2 )1[(

minus= +wait N

timeout

45

bull Most waits were for nothing ndash latch already was free

bull Latch utilization can not be high

bull Unnecessary spins provoke CPU thrashing

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682

92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE

sqlplus nolog demo_05sql 45

RDTEX | Latch internals | RUOUG Seminar

kslgetl(0x50006318 1)

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt kslges(0x50006318 ) -wait latch get

-gt skgslsgts(0x50006318 ) -spin latch get

-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT

-gt kskthbwt(0x0)

-gt kslwlmod() - set up Wait List

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt skgpwwait -sleep latch getsemop(11 17-10 1)

983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140

46

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782

RELIABLE LATCH WAITS

bull Hidden latch wait revolution that we missed Latch relies on waitposting

bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout

bull Exclusive latch spins 20000 cycles by default

bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock

bull Latches assigned to non-default class wait until timeout

bull Version 90 was transient and used mix of backoff and wait posting

This is why _LATCH_WAIT_POSTING parameter disappeared in9i

bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin

RDTEX | Latch internals | RUOUG Seminar

47

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882

IMAGINE THE TIME MICROSCOPE

| a a s | a

Reality One million zoomed time

1 us 1 sec

1 ms 17 min1 sec 115 days

Light speed (300000 kmsec) Sonic speed (300 msec)

CPU tick (2 GHz) 00005 sec

Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec

Max spin time for shared latch (~220005 ticks) ~10 sec

Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec

Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec

OS context switch time (10us-1ms) 10sec-17min

48

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982

LATCH WAITS UNDER TIME MICROSCOPE

Old 8i latch algorithmSpin for latch during 5 seconds

Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min

hellipSpin again for 5 seconds

Sleep again for 23 days (2 sec) hellip

New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)

If spin not succeeded go to sleep until post

According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)

| |

49

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082

_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES

bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and

_LATCH_CLASS_0 parameters

bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000

bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter

bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch

50

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182

TIMED LATCH WAIT POSTING

bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel

bull This can lead to instance hang ndash process will never be woken up

bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout

bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem

bull Generic AIX ports always use timed latch wait posting

bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux

51

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282

FINE GRAIN LATCH TUNING

bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting

bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7

bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo

microseconds using pollsys() (not semtimedop()) system call

ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times

Loop up to ldquoSpinsrdquo cycles

Yield CPU using yield() (or sched_yield())

ndash Sleep for ldquoSleepXrdquo usecs

ndash Then spin again hellip

52

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382

TRACES OF NONDEFAULT CLASS LATCH

983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152

991270

9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000

991270 E983158983141983150983156 10046 983156983154983137983139983141

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482

PERFORMANCE OF LATCH CLASSES

Row cache objects latch contention testcase on ODA (24 SMT)54

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582

IS THE _SPIN_COUNT REALLY STATIC

bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch

waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used

ldquocache buffers chainrdquo latch contention

bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass

(_LATCH_CLASS_X) By default ndash 20000

bull Shared latch spin can be tuned by _SPIN_COUNT value By

default ndash 2000 However this is not trivial

55

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682

SHARED LATCH BEHAVE LIKE ENQUEUE

bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80

bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests

bull This blocking achieved by special 0x40000000 bit in the latch

value This bit is a flag that some incompatible latch acquisitionis in progress

bull Queued processes will be woken up one by one on latch release

bull X mode latch gets effectively serialize the shared latch

bull See my blog for description of experiments

56

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782

TRACE OF SHARED LATCH WAIT

Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the

latch wait list and then spins again

983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141

983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156

983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983147983155983148983159983148983149983151983140()

983147983155983148983155983147983143983155(098316050009BAC )

983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140

983155983141983149983151983152(27)

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882

SHARED LATCH ACQUISITION

bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned

bull X mode shared latch get spins by default up to 4000 cycles

bull S mode does not spin at all (or spins in unknown way)

S mode get X mode get

Held in S mode Compatible 2_SPIN_COUNT

Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT

58

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982

LATCH RELEASE

bull Free the latch ndash kslfre(laddr)

bull Oracle process releases the latch nonatomically

bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process

bull This requires less bus invalidation and ensures propagation of latch

release to other local cachesbull Not fair policy Spinners on the local CPU board have the

preference

bull Then process posts the first process in the list of waiters

59

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082

LATCH CONTENTION

RDTEX | Latch internals | RUOUG Seminar60

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182

LATCH CONTENTION

bull Occurs when a latch is required by several processes at thesame time

bull Mostly due to bugs or abnormal application behavior

bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp

bull To diagnose the latch contention we need to investigate thelatch statistics [8]

61

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282

DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS

Latch requests arrival rate

Immediate gets efficiency

Latch sleeps ratio

Latch wait time per second

Latch spin efficiencymisses

getsspin∆

∆= _σ

misses

sleepsK

∆=

time

gets

∆=λ

gets

misses

∆= ρ

time

timewait W

∆=

_

Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics

62

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382

SAMPLED AVERAGES

By sampling of x$ksupr we can estimate

ndash Length of latch wait queue

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS

ndash Number of spinning processes in 10g

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS

Sampling of v$lathholder estimates latch utilization

Using DTrace we can measure

ndash Latch acquisition and holding times

ndash Actual spin count

w L

s N

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482

DERIVED LATCH STATISTICS

Latch utilization (PASTA)

Average holding time

Length of latch wait list

Recurrent sleeps ratio

Latch acquisition time

time

timeholdinglatchU

∆=asymp

__ ρ

Re_100

___

questsGet

TimeSnap MissGet Pct S

lowast==

λ

W L =

κ

κ σ 1minus+

)(1

W N T saq += minusλ

64

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 3: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 382

EPISODE OF LATCH CONTENTIONbull Oracle instance hangs due to heavy cache buffers chains latch contention

3

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 482

WHAT WE ALL KNEW ABOUT ORACLE LATCHES

bull Latches spin up to the instance parameter _SPIN_COUNT

that defaults to 2000bull This number can be tuned dynamically

bull Latch waits use increasing exponential backoff

bull Latch waits are very shortbull We should tune application not the _SPIN_COUNT

RDTEX | Latch internals | RUOUG Seminar

4

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 582

CONTEMPORARY LATCHES SINCE 92

RDTEX | Latch internals | RUOUG Seminar

5

I was wondering to find that

bull Exclusive latch spins 10 times morebull The exponential backoff was disappeared more then decade

ago in Oracle 92

bull All except one latch use wait posting now

bull Latch free wait may be infinite

bull Sometimes it is worth to tune the _SPIN_COUNT

bull This parameter is static for most latches

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 682

GREAT SOURCES OF LATCH INFO

ldquoOracle8i Internal Services for Waits Latches Locks and

Memoryrdquo by Steve Adams [1] 1999 ndash Founded new era of Oracle performance tuning

ndash However the Oracle latch technologies were evolved greatly duringthe decade

ldquoSystematic Latch Contention Troubleshootingrdquo approach byTanel Poder [2]

ndash Latchprofxsql script revolutionized the latch tuning

ldquoOracle Core Essential Internals for DBAs and Developersby Jonathan Lewis [3] 2011

RDTEX | Latch internals | RUOUG Seminar

6

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 782

REVIEW OF SERIALIZATION MECHANISMS IN ORACLE

bull Latches are simple low-level serialization mechanisms that coordinatemultiuser access to shared data structures objects and fileshellip

bull A mutual exclusion object (mutex) is a low-level mechanism thatprevents an object in memory from aging out or from being corrupted hellip

bull Internal locks are higher-level more complex mechanisms hellip

Oraclereg Database Concepts 112

bull KGX Mutexes appeared in latest Oracle versions inside Library Cache

RDTEX | Latch internals | RUOUG Seminar

A983139983139983141983155983155

A983139983153983157983145983155983145983156983145983151983150

983123983152983145983150983148983151983139983147 983138983137983155983141983140

983124983145983149983141983155983139983137983148983141

983116983145983142983141983139983161983139983148983141

983116983151983139983147983155

983123983141983158983141983154983137983148 983117983151983140983141983155

FIF983119

983118983151

gt 983117983145983148983148983145983155983141983139983151983150983140983155

D983161983150983137983149983145983139

983116983137983156983139983144983141983155

983124983161983152983141983155 983137983150983140 983117983151983140983141983155

983123I983122983119 (983155983152983145983150) + FIF983119

983129983141983155

983117983145983139983154983151983155983141983139983151983150983140983155

983123983156983137983156983145983139

983117983157983156983141983160983141983155

983119983152983141983154983137983156983145983151983150983155

983123I983122983119

983129983141983155

983123983157983138983117983145983139983154983151983155983141983139983151983150983140983155

D983161983150983137983149983145983139 7

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 882

HOW SPINLOCK WORKS

RDTEX | Latch internals | RUOUG Seminar

8

CPU3

Free FreeHolding by CPU 1

Spin polls the spinlock location

Get

Miss Sleep Spin count ( )∆

Holding time (S )

CPU2 Spin Holding the spinlock

Holding by CPU 2

CPU1 Holding the spinlock

The spinlock location

Sleep releases the CPU

bull Oracle latches

ndash Use atomic hardware instruction for Immediate get

ndash If missed process spins by polling latch location during Spin get

ndash Number of spin cycles is bounded by spin count

ndash In spin get not succeed the process acquiring the latch sleeps

hellip

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 982

CLASSIC SPINLOCKS

bull Wiki hellip spinlock hellip waits in a loop repeatedly checking until the lockbecomes available hellip

bull Introduced by Edsger Dijkstra [4] in 1965 Have been thoroughlyinvestigated since that time [5]

bull Many sophisticated spinlock realizations were proposed and evaluated

(TS TTS Delay MCS Anderson)

bull Two general types

ndash System spinlocks Kernel OS threads cannot wait Major metrics

atomic operations frequency Shared bus utilization

ndash User spinlocks Oracle latch and mutex Average lock holding time ~10 us It is more efficient to poll a lock rather than pre-empt the threaddoing 1 ms context switch Metrics CPU and elapsed times

RDTEX | Latch internals | RUOUG Seminar

9

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1082

THE NEED FOR SPIN

RDTEX | Latch internals | RUOUG Seminar

10

Time to complete CBC latch contention testcase vs number of threads

No spinning _spin_count=0

_spin_count=2000

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1182

SPINLOCK REALIZATIONS

RDTEX | Latch internals | RUOUG Seminar

Spinlock Pseudocode Problems

TS

pre-112 mutex

while(Test_and_Set(lock)) Bus saturation

TTS

Oracle latch

while(lock||Test_and_Set(lock)) Invalidation storms onrelease (ldquoopen doorrdquo

ldquothundering herdsrdquo)

Delay

Latch and 112

Mutex

Adjustable delay Slower under contention

Anderson MCSetc

Queues Widely used in JavaLinux kernel hellip

CPU and memoryoverhead preemptionissues

11

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1282

THE TOOLS

RDTEX | Latch internals | RUOUG Seminar12

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1382

DTRACE SOLARIS DYNAMIC TRACING FRAMEWORK

DTrace allows us to investigate how Oracle latches perform in real time ndash Create triggers on any event inside Solaris and function call inside Oracle

providermodulefunctionname

pid1910oraclekslgetlentry ndash Write trigger bodies ndash actions May read and change any address location ndash Can count the latch spins trace the latch waits perform experiments ndash Measure times and distributions up to microseconds precision

RDTEX | Latch internals | RUOUG Seminar

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313500

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1482

ORADEBUG INTERNAL ORACLE DEBUGGER

RDTEX | Latch internals | RUOUG Seminar

bull oradebug call Allows us invoke any internal Oracle function manually

SQLgt oradebug peek 200222A0 24

[200222A0 200222B8) = 00000016 00000001 000001D0 00000007

SQLgt oradebug call kslgetl 0x5001A020 1 2 3Function returned 1

bull oradebug peek Examines contents of any memory location

bull oradebug poke Modifies memory

bull oradebug watch Watch a region of memory

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1582

LATCH CONTENTION TESTCASES

RDTEX | Latch internals | RUOUG Seminar

15

create table latch_contention as select from dualhellipfor i in 11000000loopselect dummy into a from latch_contention

end loop

bull Exclusive and shared latches behave differentlybull Contention for shared cache buffer chains latch arises when manysessions scan the same block concurrently and _fastpin_enable=0

bull To induce the contention for exclusive ldquorow cache objectsrdquo latch protectingdc_users row cache just add several dynamic RLS policies

exec dbms_rlsadd_policy(systemlatch_contentionrls_1 hellip

exec dbms_rlsadd_policy(systemlatch_contentionrls_2 helliphellip

bull These dummy policies generate predicates 1=1bull Many other contention scenarios possible

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313509831351

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1682

ROW CACHE LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

16

Elapsed and CPU times vs number of threadsOracle Database Appliance (24 SMT)

Latch wait

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1782

HOW ORACLEREQUESTS THE LATCH

RDTEX | Latch internals | RUOUG Seminar17

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1882

DTRACE REVEALS LATCH INTERFACE ROUTINES

Oracle calls the following functions to acquire the latch

ndash kslgetl(laddr wait why where) - get exclusive latch ndash kslg2c(l1l2trcwhy where) - get two excl child latches

ndash kslgetsl (laddrwaitwhywheremode) - get shared latch

bull 11g - ksl_get_shared_latch (hellip)

ndash kslg2cs(l1l2modetrcwhy where)) - get two shr child latches

ndash kslgpl(laddrcommentwhywhere) - get parent and all childs

ndash kslfre(laddr) - free the latch

Oracle give us possibility to do the same by oradebug call

RDTEX | Latch internals | RUOUG Seminar

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313501

983125983155983141 32983085983138983145983156 983152983151983154983156983155

64983138983145983156 O983154983137983139983148983141 983139983151983154983154983157983152983156983155 983156983144983141 983137983154983143983157983149983141983150983156983155 983151983142 983151983154983137983140983141983138983157983143 983139983137983148983148 983145983150 10204 983137983150983140 983137983138983151983158983141

18

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1982

ldquoWHERErdquo AND ldquoWHYrdquo ORACLE GETS THE LATCH

To request the latch Oracle kernel routines need

bull laddres ndash address of latch in SGAbull wait ndash flag for no-wait (0) or wait (1) latch acquisition

bull where ndash integer code for location from where the latch is acquired

Oracle lists possible latch ldquowhererdquo values in x$ksllwindxbull why - context of why the latch is acquiring at this ldquowhererdquo It

contains DBA for CBC latches (Tanel Poder) SGA chunk addressfor shared pool latch session address etchellip ldquoWhyrdquo meaning for

some ldquowhererdquo can be found in x$ksllwksllwlbl

bull mode ndash requesting state for shared lathes

ndash 8 SHARED mode 16 EXCLUSIVE mode

RDTEX | Latch internals | RUOUG Seminar

19

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2082

LIST OF LATCH ldquoWHERErdquo LOCATIONS

X$KSLLW ndash List of latch [W]here Part of v$latch_misses

ndash ksllwnam ndash Kernel function name(ldquokghupr1rdquo ldquoksqdelrdquo etc) ndash ksllwlbl ndash Description of ldquowhyrdquo context

Join this code on indx to

ndash x$ksuprlatksulawhr (v$latchholder)ndash ldquowhererdquo value that wasused by latch holder

ndash x$kslltkslltwhr (v$latch) ndash ldquowhererdquo that was used by lastsuccessful latch acquisition

ndash x$ksuprksllawer (v$process) ndash ldquowhererdquo that was used byprocess is being waiting for latch

ndash x$kslwscindx (v$latch_misses) ndash latch sleep statistics by

ldquowhererdquo

RDTEX | Latch internals | RUOUG Seminar

20

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2182

V$LATCH_MISSES

bull Based on X$KSLWSC - No[W]ait and [S]leep [C]ount stats byContext Documentation is rather obscure

bull The following counters are incremented on each latch sleep

sleep_count (ldquowhererdquo the latch was held)

wtr_slp_count (ldquowhererdquo the latch get was requested)

bull nwfail_count is incremented only for session which SID is equalto _LATCH_MISS_STAT_SID parameter

bull longhold_count is incremented for shared latches only when

someone held the latch at this ldquowhererdquo for the entire duration ofsleep

RDTEX | Latch internals | RUOUG Seminar

21

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2282

LATCH IS HOLDING BY PROCESS NOT SESSION

bull

RDTEX | Latch internals | RUOUG Seminar

983123983156983154983157983139983156 983147983155983157983152983154 983163

991270

983123983156983154983157983139983156 983147983155983148983148983137983163983147983155983148983148983156 98314798315598314898314898313798314898313798315698313114983133

983165

991270983165

983120983154983151983139983141983155983155 983142983145983160983141983140 983137983154983154983137983161

983158$983152983154983151983139983141983155983155 983085gt 983160$983147983155983157983152983154

983123983156983154983157983139983156 983147983155983148983148983156983163

991270

983165

L983145983155983156 983151983142 983137983148983148 983148983137983156983139983144983141983155

983158$983148983137983156983139983144 983085gt983160$983147983155983148983148983156

Each process has an array of references to the latches it is holding

22

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2382

KSLLA IS HIDDEN IN V$PROCESS

bull The process latching info is the kslla structure embedded in theprocess state object

bull Some kslla fields were externalized to SQL

bull kslla contains array of latches holding by process at each level andinformation about the state of any current attempt to acquire a

latch Not externalized directly

bull When you query v$latchholder x$ksuprlat Oracle scansv$processx$ksupr in order to find all the processes that are

holding the latchesbull Tanel Poder uses high frequency sampling of v$latchholder to

systematically troubleshoot latch contention

RDTEX | Latch internals | RUOUG Seminar

23

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2482

X$KSUPRKSLLA FIELDS INSTRUMENT THE LATCH GET

bull ksllalaq ndash address of latch acquiring Populated during immediateget (and spin before 11g)

bull ksllawat - latch being waited for This is v$processlatchwait

bull ksllawhy ndash ldquowhyrdquo for the latch being waited for

bull ksllawere ndash ldquowhererdquo for the latch being waited for

bull ksllalow ndash bit array of levels of currently holding latches

bull ksllaspn - latch this process is spinning on v$processlatchspinNot populated since 81

bull ksllaps - inter-process post statistics

RDTEX | Latch internals | RUOUG Seminar

24

Generic ldquolatch freerdquo and 27 latch specific ldquolatch free rdquo wait events areassociated with sleep only Spin is invisible to Oracle Wait Interface

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2582

THE LATCH STRUCTURE ndash KSLLT

RDTEX | Latch internals | RUOUG Seminar

struct ksllt

ltLatchgt

ldquowhererdquo and ldquowhyrdquo

Level latch class other attributes

Statistics

Latch wait list header

hellip

25

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2682

LATCH SIZE BY VERSION

nix 32bit nix 64bit Windows 32bit

734 92 - 120

806 104 - 104817 104 144 104

901 200 160

920 196 240 200

1010 256 208

1020 - 11203 100 160 104

RDTEX | Latch internals | RUOUG Seminar

x$ksmfsv ndash list of all fixed SGA variables983123E983116EC983124 DI983123983124I983118C983124 983147983155983149983142983155983155983145983162

F983122983119983117 983160$983147983155983149983142983155983158

983127HE983122E 983147983155983149983142983155983156983161983152 = 983147983155983148983148983156

Latch structure was bigger in 101 due to additional latch statistics

26

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2782

ORACLE LATCH IS NOT JUST A SINGLE MEMORY LOCATION

bull Before 11g Value of first latch byte (word for shared latches) wasused to determine latch state

0x00 ndash latch is free

0xFF ndash exclusive latch is busy Was 0x01 in Oracle 7

0x010x02hellip - shared latch holding by 12 hellip processes simultaneously

0x20000000 | pid - shared latch holding exclusively

bull In 11g first latch word shows the pid of the latch holder

0x00 ndash latch is free0x12 ndash Oracle process with pid 18 holds the exclusive latch

RDTEX | Latch internals | RUOUG Seminar

27

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2882

BUSY ORACLE 11G LATCH IN MEMORY

bull 112 Exclusive latch (32-bit platform)

bull 112 Shared latch EXCLUSIVE mode

RDTEX | Latch internals | RUOUG Seminar

SQLgt oradebug peek 200222A0 24

[200222A0 200222B8) = 00000016 00000001 000001D0 00000007

pid^ gets latch level

SQLgt oradebug peek 0xampladdr 24

[2000C814 2000C82C) = 2000001F 0000007C 0000007E 00000006 pid^ gets latch level

983139983151983150983156 983155983153983148983152983148983157983155 983150983151983148983151983143 9831409831419831499831519831350128

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2982

LATCH FIXED TABLES

bull Before Oracle 11g all the latches were externalized to SQL inv$latch (x$ksllt)

bull In 11g parent (fixed SGA) latches are externalized inv$latch_parent (x$kslltr_parent)

bull In 11g child (allocated on startup) latches are externalized in

v$latch_children (x$kslltr_children) bull Oracle 11g v$latch (x$kslltr) merges these data Beware select

from x$kslltr acquires all parent latches and their children

bull Only part of latch structure is visible in x$ksllthellip fixed tables ndash kslltwhr kslltwhy ndash ldquowhererdquo and ldquowhyrdquo latch was acquired last time

ndash kslltwgt hellip - statistics

ndash kslltlvl hellip - attributes

ndash kslltwtt ndash latch wait time

RDTEX | Latch internals | RUOUG Seminar

29

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3082

CUMULATIVE V$LATCH STATISTIC COUNTERS

RDTEX | Latch internals | RUOUG Seminar

Statistics x$ksllt When and how it increments

GETS kslltwgt ldquo++rdquo after wait mode latch get

MISSES kslltwff ldquo++rdquo after wait get if it was missed

SLEEPS kslltwsl ldquo+number_of_sleepsrdquo during get

SPIN_GETS ksllthst0 ldquo++rdquo if get was missed but not slept

WAIT_TIME kslltwtt ldquo+wait_timerdquo after latch get

IMMEDIATE_GETS kslltngt ldquo++rdquo after nowait mode latch get Is notincremented on miss

IMMEDIATE_MISSES kslltnfa ldquo++rdquo if nowait mode get was missed

30

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3182

MANY STATISTICS WERE LOST IN 102

In 102 Oracle made obsolete many useful latch statistics

Latch size was decreased and the following fields always contain 0now in v$latch

ndash v$latchwaiters_woken

ndash v$latchwaits_holding_latch

ndash v$latchsleep1hellipsleep10

ndash and many undocumented statistics

This was mentioned in 102 Reference

RDTEX | Latch internals | RUOUG Seminar

31

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3282

X$KSLLD - LATCH DESCRIPTORS

bull Flags and attributes describing each latch are stored in kslldt arrayof kslld structures

bull [K]ernel [S]ervice [L]atch [L]ock [D]escriptor

bull Some but not all of kslld fields are externalized via x$kslld andv$latchname

bull Located in the PGA

RDTEX | Latch internals | RUOUG Seminar

32

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382

LATCH ATTRIBUTES

bull Each latch have at least the following attributes in kslldt

ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared

ndash PAR Is the latch Solitary or Parent for the family of child latches

ndash G2C Can two child latches be simultaneously requested in wait mode

using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92

ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above

ndash Level 0-14 Used to prevent latch deadlocks

ndash Class 0-7 Spin and wait class assigned to the latch 92 and above

RDTEX | Latch internals | RUOUG Seminar

33

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482

LATCH ATTRIBUTES BY ORACLE VERSION

RDTEX | Latch internals | RUOUG Seminar

11203 553 154 72 - 6 93

Oracle Number of latches PAR G2C LNG UFS SHARED

7340 53 14 2 3 - -

8063 80 21 7 3 - 38174 152 48 19 4 - 9

9208 242 79 37 - - 19

10202 385 114 55 - 4 47

10203 388 117 58 - 4 48

10204 394 117 59 - 4 50

11106 496 145 67 - 6 81

11107 502 145 67 - 6 83

11201 535 149 70 - 6 86

11202 551 153 72 - 6 91

34

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582

LATCH LEVELS

bull Every latch has a level in the range 0-14 to avoid deadlocks

bull Process can request a latch X in wait mode after obtaining latch Y if and

only if level X gt level Y

bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y

bull If these rules are broken the Oracle process raises

ORA-600[504] - trying to obtain another latch at incompatible level Note 281041

ORA-600[525] - trying to obtain another latch at the same level Note 1388881

ORA-600[526] - latch child order violation Note 1388891

RDTEX | Latch internals | RUOUG Seminar

35

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682

PARENT AND CHILD LATCHES

bull Latch can be a parent child or solitary latch

bull Both parent and child latches share the same latch name

bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss

bull Child latches are allocated during instance startup

bull The parent latch can be gotten independently

bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s

RDTEX | Latch internals | RUOUG Seminar

(latch info) wait_event=0 bits=10

holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl

Context saved from call 0

state=busy

36

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782

LATCH TREES

ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches

Direct SGA access program output for 9206 instance with too smallshared pool

RDTEX | Latch internals | RUOUG Seminar

ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1

waiter ospid 129 sid 72 pid 45

holding a154b7120 library cache (157) level=5 child=17 kglupc child

waiter ospid 18255 sid 65 pid 930

waiter ospid 6690 sid 554 pid 1654

waiter ospid 4685 sid 879 pid 1034

hellip

waiter ospid 29749 sid 180 pid 155

holding a154b7db8 library cache (157) level=5 child=4 kglupc child

waiter ospid 13104 sid 281 pid 220

waiter ospid 24089 sid 565 pid 636

waiter ospid 25002 sid 621 pid 1481

waiter ospid 16930 sid 1046 pid 783hellip

37

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882

LATCH CLASSES

bull Classes with different spin and wait policies

bull Up to 8 classes By default all the latches belong to class 0

bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2

bull Latch can be assigned to class by its number For example

ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000

ndash _LATCH_CLASSES=46

bull Will be discussed later

RDTEX | Latch internals | RUOUG Seminar

SQLgt select from x$ksllclass

INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4

---- ------ ----- -------- ------ ------ ------ ------ ------

0 20000 0 1 8000 8000 8000 8000 8000

hellip

6 100 12 1000 2000 3000 4000 4000 4000

38

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982

LATCH ACQUISITION IN NO-WAIT MODE

bull kslgetl(laddr0whywhere) ndash Exclusive latch get

bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep

ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches

ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches

bull IMMEDIATE_ statistics are updated

bull Used when many candidate latches are available to bypass the

latch level rule or as part of CBC chain examination

RDTEX | Latch internals | RUOUG Seminar

sqlplus nolog demo_03sql39

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082

LATCH WAITS

RDTEX | Latch internals | RUOUG Seminar 40

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182

WAITING FOR THE LATCH

RDTEX | Latch internals | RUOUG Seminar

983123 G A

C983120983125 1 C983120983125 2

983120983154983151983139983141983155983155 A

983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)

983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144

983116983137983156983139983144

983120983154983151983139983141983155983155 B

41

RDTEX | L h i l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282

LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION

bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries

again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of

times

bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep

bull A sleep corresponds to a wait for the latch free wait event

bull Initially a process sleeps for 001 s This time is doubled in every

subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip

RDTEX | Latch internals | RUOUG Seminar

Still did not found any versionwhich used this

May be it was lt= Oracle 6

42

RDTEX | L t h i t l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382

LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81

Latch gets with wait (kslgetl(laddress1hellip))

ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times

ndash Sleep on latch free event with exponential backoff

ndash Repeat

RDTEX | Latch internals | RUOUG Seminar

43

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482

8I LATCH SPIN CODE FLOW USING DTRACE

RDTEX | Latch internals | RUOUG Seminar

bull hellip Event 10046 trace

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2

983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29

983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270

sqlplus nolog demo_04_8sql 29

44

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582

POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF

bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec

RDTEX | Latch internals | RUOUG Seminar

12 ]2 )1[(

minus= +wait N

timeout

45

bull Most waits were for nothing ndash latch already was free

bull Latch utilization can not be high

bull Unnecessary spins provoke CPU thrashing

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682

92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE

sqlplus nolog demo_05sql 45

RDTEX | Latch internals | RUOUG Seminar

kslgetl(0x50006318 1)

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt kslges(0x50006318 ) -wait latch get

-gt skgslsgts(0x50006318 ) -spin latch get

-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT

-gt kskthbwt(0x0)

-gt kslwlmod() - set up Wait List

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt skgpwwait -sleep latch getsemop(11 17-10 1)

983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140

46

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782

RELIABLE LATCH WAITS

bull Hidden latch wait revolution that we missed Latch relies on waitposting

bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout

bull Exclusive latch spins 20000 cycles by default

bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock

bull Latches assigned to non-default class wait until timeout

bull Version 90 was transient and used mix of backoff and wait posting

This is why _LATCH_WAIT_POSTING parameter disappeared in9i

bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin

RDTEX | Latch internals | RUOUG Seminar

47

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882

IMAGINE THE TIME MICROSCOPE

| a a s | a

Reality One million zoomed time

1 us 1 sec

1 ms 17 min1 sec 115 days

Light speed (300000 kmsec) Sonic speed (300 msec)

CPU tick (2 GHz) 00005 sec

Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec

Max spin time for shared latch (~220005 ticks) ~10 sec

Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec

Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec

OS context switch time (10us-1ms) 10sec-17min

48

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982

LATCH WAITS UNDER TIME MICROSCOPE

Old 8i latch algorithmSpin for latch during 5 seconds

Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min

hellipSpin again for 5 seconds

Sleep again for 23 days (2 sec) hellip

New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)

If spin not succeeded go to sleep until post

According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)

| |

49

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082

_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES

bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and

_LATCH_CLASS_0 parameters

bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000

bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter

bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch

50

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182

TIMED LATCH WAIT POSTING

bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel

bull This can lead to instance hang ndash process will never be woken up

bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout

bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem

bull Generic AIX ports always use timed latch wait posting

bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux

51

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282

FINE GRAIN LATCH TUNING

bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting

bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7

bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo

microseconds using pollsys() (not semtimedop()) system call

ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times

Loop up to ldquoSpinsrdquo cycles

Yield CPU using yield() (or sched_yield())

ndash Sleep for ldquoSleepXrdquo usecs

ndash Then spin again hellip

52

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382

TRACES OF NONDEFAULT CLASS LATCH

983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152

991270

9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000

991270 E983158983141983150983156 10046 983156983154983137983139983141

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482

PERFORMANCE OF LATCH CLASSES

Row cache objects latch contention testcase on ODA (24 SMT)54

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582

IS THE _SPIN_COUNT REALLY STATIC

bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch

waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used

ldquocache buffers chainrdquo latch contention

bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass

(_LATCH_CLASS_X) By default ndash 20000

bull Shared latch spin can be tuned by _SPIN_COUNT value By

default ndash 2000 However this is not trivial

55

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682

SHARED LATCH BEHAVE LIKE ENQUEUE

bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80

bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests

bull This blocking achieved by special 0x40000000 bit in the latch

value This bit is a flag that some incompatible latch acquisitionis in progress

bull Queued processes will be woken up one by one on latch release

bull X mode latch gets effectively serialize the shared latch

bull See my blog for description of experiments

56

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782

TRACE OF SHARED LATCH WAIT

Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the

latch wait list and then spins again

983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141

983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156

983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983147983155983148983159983148983149983151983140()

983147983155983148983155983147983143983155(098316050009BAC )

983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140

983155983141983149983151983152(27)

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882

SHARED LATCH ACQUISITION

bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned

bull X mode shared latch get spins by default up to 4000 cycles

bull S mode does not spin at all (or spins in unknown way)

S mode get X mode get

Held in S mode Compatible 2_SPIN_COUNT

Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT

58

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982

LATCH RELEASE

bull Free the latch ndash kslfre(laddr)

bull Oracle process releases the latch nonatomically

bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process

bull This requires less bus invalidation and ensures propagation of latch

release to other local cachesbull Not fair policy Spinners on the local CPU board have the

preference

bull Then process posts the first process in the list of waiters

59

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082

LATCH CONTENTION

RDTEX | Latch internals | RUOUG Seminar60

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182

LATCH CONTENTION

bull Occurs when a latch is required by several processes at thesame time

bull Mostly due to bugs or abnormal application behavior

bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp

bull To diagnose the latch contention we need to investigate thelatch statistics [8]

61

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282

DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS

Latch requests arrival rate

Immediate gets efficiency

Latch sleeps ratio

Latch wait time per second

Latch spin efficiencymisses

getsspin∆

∆= _σ

misses

sleepsK

∆=

time

gets

∆=λ

gets

misses

∆= ρ

time

timewait W

∆=

_

Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics

62

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382

SAMPLED AVERAGES

By sampling of x$ksupr we can estimate

ndash Length of latch wait queue

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS

ndash Number of spinning processes in 10g

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS

Sampling of v$lathholder estimates latch utilization

Using DTrace we can measure

ndash Latch acquisition and holding times

ndash Actual spin count

w L

s N

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482

DERIVED LATCH STATISTICS

Latch utilization (PASTA)

Average holding time

Length of latch wait list

Recurrent sleeps ratio

Latch acquisition time

time

timeholdinglatchU

∆=asymp

__ ρ

Re_100

___

questsGet

TimeSnap MissGet Pct S

lowast==

λ

W L =

κ

κ σ 1minus+

)(1

W N T saq += minusλ

64

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 4: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 482

WHAT WE ALL KNEW ABOUT ORACLE LATCHES

bull Latches spin up to the instance parameter _SPIN_COUNT

that defaults to 2000bull This number can be tuned dynamically

bull Latch waits use increasing exponential backoff

bull Latch waits are very shortbull We should tune application not the _SPIN_COUNT

RDTEX | Latch internals | RUOUG Seminar

4

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 582

CONTEMPORARY LATCHES SINCE 92

RDTEX | Latch internals | RUOUG Seminar

5

I was wondering to find that

bull Exclusive latch spins 10 times morebull The exponential backoff was disappeared more then decade

ago in Oracle 92

bull All except one latch use wait posting now

bull Latch free wait may be infinite

bull Sometimes it is worth to tune the _SPIN_COUNT

bull This parameter is static for most latches

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 682

GREAT SOURCES OF LATCH INFO

ldquoOracle8i Internal Services for Waits Latches Locks and

Memoryrdquo by Steve Adams [1] 1999 ndash Founded new era of Oracle performance tuning

ndash However the Oracle latch technologies were evolved greatly duringthe decade

ldquoSystematic Latch Contention Troubleshootingrdquo approach byTanel Poder [2]

ndash Latchprofxsql script revolutionized the latch tuning

ldquoOracle Core Essential Internals for DBAs and Developersby Jonathan Lewis [3] 2011

RDTEX | Latch internals | RUOUG Seminar

6

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 782

REVIEW OF SERIALIZATION MECHANISMS IN ORACLE

bull Latches are simple low-level serialization mechanisms that coordinatemultiuser access to shared data structures objects and fileshellip

bull A mutual exclusion object (mutex) is a low-level mechanism thatprevents an object in memory from aging out or from being corrupted hellip

bull Internal locks are higher-level more complex mechanisms hellip

Oraclereg Database Concepts 112

bull KGX Mutexes appeared in latest Oracle versions inside Library Cache

RDTEX | Latch internals | RUOUG Seminar

A983139983139983141983155983155

A983139983153983157983145983155983145983156983145983151983150

983123983152983145983150983148983151983139983147 983138983137983155983141983140

983124983145983149983141983155983139983137983148983141

983116983145983142983141983139983161983139983148983141

983116983151983139983147983155

983123983141983158983141983154983137983148 983117983151983140983141983155

FIF983119

983118983151

gt 983117983145983148983148983145983155983141983139983151983150983140983155

D983161983150983137983149983145983139

983116983137983156983139983144983141983155

983124983161983152983141983155 983137983150983140 983117983151983140983141983155

983123I983122983119 (983155983152983145983150) + FIF983119

983129983141983155

983117983145983139983154983151983155983141983139983151983150983140983155

983123983156983137983156983145983139

983117983157983156983141983160983141983155

983119983152983141983154983137983156983145983151983150983155

983123I983122983119

983129983141983155

983123983157983138983117983145983139983154983151983155983141983139983151983150983140983155

D983161983150983137983149983145983139 7

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 882

HOW SPINLOCK WORKS

RDTEX | Latch internals | RUOUG Seminar

8

CPU3

Free FreeHolding by CPU 1

Spin polls the spinlock location

Get

Miss Sleep Spin count ( )∆

Holding time (S )

CPU2 Spin Holding the spinlock

Holding by CPU 2

CPU1 Holding the spinlock

The spinlock location

Sleep releases the CPU

bull Oracle latches

ndash Use atomic hardware instruction for Immediate get

ndash If missed process spins by polling latch location during Spin get

ndash Number of spin cycles is bounded by spin count

ndash In spin get not succeed the process acquiring the latch sleeps

hellip

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 982

CLASSIC SPINLOCKS

bull Wiki hellip spinlock hellip waits in a loop repeatedly checking until the lockbecomes available hellip

bull Introduced by Edsger Dijkstra [4] in 1965 Have been thoroughlyinvestigated since that time [5]

bull Many sophisticated spinlock realizations were proposed and evaluated

(TS TTS Delay MCS Anderson)

bull Two general types

ndash System spinlocks Kernel OS threads cannot wait Major metrics

atomic operations frequency Shared bus utilization

ndash User spinlocks Oracle latch and mutex Average lock holding time ~10 us It is more efficient to poll a lock rather than pre-empt the threaddoing 1 ms context switch Metrics CPU and elapsed times

RDTEX | Latch internals | RUOUG Seminar

9

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1082

THE NEED FOR SPIN

RDTEX | Latch internals | RUOUG Seminar

10

Time to complete CBC latch contention testcase vs number of threads

No spinning _spin_count=0

_spin_count=2000

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1182

SPINLOCK REALIZATIONS

RDTEX | Latch internals | RUOUG Seminar

Spinlock Pseudocode Problems

TS

pre-112 mutex

while(Test_and_Set(lock)) Bus saturation

TTS

Oracle latch

while(lock||Test_and_Set(lock)) Invalidation storms onrelease (ldquoopen doorrdquo

ldquothundering herdsrdquo)

Delay

Latch and 112

Mutex

Adjustable delay Slower under contention

Anderson MCSetc

Queues Widely used in JavaLinux kernel hellip

CPU and memoryoverhead preemptionissues

11

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1282

THE TOOLS

RDTEX | Latch internals | RUOUG Seminar12

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1382

DTRACE SOLARIS DYNAMIC TRACING FRAMEWORK

DTrace allows us to investigate how Oracle latches perform in real time ndash Create triggers on any event inside Solaris and function call inside Oracle

providermodulefunctionname

pid1910oraclekslgetlentry ndash Write trigger bodies ndash actions May read and change any address location ndash Can count the latch spins trace the latch waits perform experiments ndash Measure times and distributions up to microseconds precision

RDTEX | Latch internals | RUOUG Seminar

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313500

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1482

ORADEBUG INTERNAL ORACLE DEBUGGER

RDTEX | Latch internals | RUOUG Seminar

bull oradebug call Allows us invoke any internal Oracle function manually

SQLgt oradebug peek 200222A0 24

[200222A0 200222B8) = 00000016 00000001 000001D0 00000007

SQLgt oradebug call kslgetl 0x5001A020 1 2 3Function returned 1

bull oradebug peek Examines contents of any memory location

bull oradebug poke Modifies memory

bull oradebug watch Watch a region of memory

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1582

LATCH CONTENTION TESTCASES

RDTEX | Latch internals | RUOUG Seminar

15

create table latch_contention as select from dualhellipfor i in 11000000loopselect dummy into a from latch_contention

end loop

bull Exclusive and shared latches behave differentlybull Contention for shared cache buffer chains latch arises when manysessions scan the same block concurrently and _fastpin_enable=0

bull To induce the contention for exclusive ldquorow cache objectsrdquo latch protectingdc_users row cache just add several dynamic RLS policies

exec dbms_rlsadd_policy(systemlatch_contentionrls_1 hellip

exec dbms_rlsadd_policy(systemlatch_contentionrls_2 helliphellip

bull These dummy policies generate predicates 1=1bull Many other contention scenarios possible

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313509831351

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1682

ROW CACHE LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

16

Elapsed and CPU times vs number of threadsOracle Database Appliance (24 SMT)

Latch wait

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1782

HOW ORACLEREQUESTS THE LATCH

RDTEX | Latch internals | RUOUG Seminar17

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1882

DTRACE REVEALS LATCH INTERFACE ROUTINES

Oracle calls the following functions to acquire the latch

ndash kslgetl(laddr wait why where) - get exclusive latch ndash kslg2c(l1l2trcwhy where) - get two excl child latches

ndash kslgetsl (laddrwaitwhywheremode) - get shared latch

bull 11g - ksl_get_shared_latch (hellip)

ndash kslg2cs(l1l2modetrcwhy where)) - get two shr child latches

ndash kslgpl(laddrcommentwhywhere) - get parent and all childs

ndash kslfre(laddr) - free the latch

Oracle give us possibility to do the same by oradebug call

RDTEX | Latch internals | RUOUG Seminar

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313501

983125983155983141 32983085983138983145983156 983152983151983154983156983155

64983138983145983156 O983154983137983139983148983141 983139983151983154983154983157983152983156983155 983156983144983141 983137983154983143983157983149983141983150983156983155 983151983142 983151983154983137983140983141983138983157983143 983139983137983148983148 983145983150 10204 983137983150983140 983137983138983151983158983141

18

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1982

ldquoWHERErdquo AND ldquoWHYrdquo ORACLE GETS THE LATCH

To request the latch Oracle kernel routines need

bull laddres ndash address of latch in SGAbull wait ndash flag for no-wait (0) or wait (1) latch acquisition

bull where ndash integer code for location from where the latch is acquired

Oracle lists possible latch ldquowhererdquo values in x$ksllwindxbull why - context of why the latch is acquiring at this ldquowhererdquo It

contains DBA for CBC latches (Tanel Poder) SGA chunk addressfor shared pool latch session address etchellip ldquoWhyrdquo meaning for

some ldquowhererdquo can be found in x$ksllwksllwlbl

bull mode ndash requesting state for shared lathes

ndash 8 SHARED mode 16 EXCLUSIVE mode

RDTEX | Latch internals | RUOUG Seminar

19

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2082

LIST OF LATCH ldquoWHERErdquo LOCATIONS

X$KSLLW ndash List of latch [W]here Part of v$latch_misses

ndash ksllwnam ndash Kernel function name(ldquokghupr1rdquo ldquoksqdelrdquo etc) ndash ksllwlbl ndash Description of ldquowhyrdquo context

Join this code on indx to

ndash x$ksuprlatksulawhr (v$latchholder)ndash ldquowhererdquo value that wasused by latch holder

ndash x$kslltkslltwhr (v$latch) ndash ldquowhererdquo that was used by lastsuccessful latch acquisition

ndash x$ksuprksllawer (v$process) ndash ldquowhererdquo that was used byprocess is being waiting for latch

ndash x$kslwscindx (v$latch_misses) ndash latch sleep statistics by

ldquowhererdquo

RDTEX | Latch internals | RUOUG Seminar

20

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2182

V$LATCH_MISSES

bull Based on X$KSLWSC - No[W]ait and [S]leep [C]ount stats byContext Documentation is rather obscure

bull The following counters are incremented on each latch sleep

sleep_count (ldquowhererdquo the latch was held)

wtr_slp_count (ldquowhererdquo the latch get was requested)

bull nwfail_count is incremented only for session which SID is equalto _LATCH_MISS_STAT_SID parameter

bull longhold_count is incremented for shared latches only when

someone held the latch at this ldquowhererdquo for the entire duration ofsleep

RDTEX | Latch internals | RUOUG Seminar

21

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2282

LATCH IS HOLDING BY PROCESS NOT SESSION

bull

RDTEX | Latch internals | RUOUG Seminar

983123983156983154983157983139983156 983147983155983157983152983154 983163

991270

983123983156983154983157983139983156 983147983155983148983148983137983163983147983155983148983148983156 98314798315598314898314898313798314898313798315698313114983133

983165

991270983165

983120983154983151983139983141983155983155 983142983145983160983141983140 983137983154983154983137983161

983158$983152983154983151983139983141983155983155 983085gt 983160$983147983155983157983152983154

983123983156983154983157983139983156 983147983155983148983148983156983163

991270

983165

L983145983155983156 983151983142 983137983148983148 983148983137983156983139983144983141983155

983158$983148983137983156983139983144 983085gt983160$983147983155983148983148983156

Each process has an array of references to the latches it is holding

22

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2382

KSLLA IS HIDDEN IN V$PROCESS

bull The process latching info is the kslla structure embedded in theprocess state object

bull Some kslla fields were externalized to SQL

bull kslla contains array of latches holding by process at each level andinformation about the state of any current attempt to acquire a

latch Not externalized directly

bull When you query v$latchholder x$ksuprlat Oracle scansv$processx$ksupr in order to find all the processes that are

holding the latchesbull Tanel Poder uses high frequency sampling of v$latchholder to

systematically troubleshoot latch contention

RDTEX | Latch internals | RUOUG Seminar

23

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2482

X$KSUPRKSLLA FIELDS INSTRUMENT THE LATCH GET

bull ksllalaq ndash address of latch acquiring Populated during immediateget (and spin before 11g)

bull ksllawat - latch being waited for This is v$processlatchwait

bull ksllawhy ndash ldquowhyrdquo for the latch being waited for

bull ksllawere ndash ldquowhererdquo for the latch being waited for

bull ksllalow ndash bit array of levels of currently holding latches

bull ksllaspn - latch this process is spinning on v$processlatchspinNot populated since 81

bull ksllaps - inter-process post statistics

RDTEX | Latch internals | RUOUG Seminar

24

Generic ldquolatch freerdquo and 27 latch specific ldquolatch free rdquo wait events areassociated with sleep only Spin is invisible to Oracle Wait Interface

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2582

THE LATCH STRUCTURE ndash KSLLT

RDTEX | Latch internals | RUOUG Seminar

struct ksllt

ltLatchgt

ldquowhererdquo and ldquowhyrdquo

Level latch class other attributes

Statistics

Latch wait list header

hellip

25

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2682

LATCH SIZE BY VERSION

nix 32bit nix 64bit Windows 32bit

734 92 - 120

806 104 - 104817 104 144 104

901 200 160

920 196 240 200

1010 256 208

1020 - 11203 100 160 104

RDTEX | Latch internals | RUOUG Seminar

x$ksmfsv ndash list of all fixed SGA variables983123E983116EC983124 DI983123983124I983118C983124 983147983155983149983142983155983155983145983162

F983122983119983117 983160$983147983155983149983142983155983158

983127HE983122E 983147983155983149983142983155983156983161983152 = 983147983155983148983148983156

Latch structure was bigger in 101 due to additional latch statistics

26

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2782

ORACLE LATCH IS NOT JUST A SINGLE MEMORY LOCATION

bull Before 11g Value of first latch byte (word for shared latches) wasused to determine latch state

0x00 ndash latch is free

0xFF ndash exclusive latch is busy Was 0x01 in Oracle 7

0x010x02hellip - shared latch holding by 12 hellip processes simultaneously

0x20000000 | pid - shared latch holding exclusively

bull In 11g first latch word shows the pid of the latch holder

0x00 ndash latch is free0x12 ndash Oracle process with pid 18 holds the exclusive latch

RDTEX | Latch internals | RUOUG Seminar

27

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2882

BUSY ORACLE 11G LATCH IN MEMORY

bull 112 Exclusive latch (32-bit platform)

bull 112 Shared latch EXCLUSIVE mode

RDTEX | Latch internals | RUOUG Seminar

SQLgt oradebug peek 200222A0 24

[200222A0 200222B8) = 00000016 00000001 000001D0 00000007

pid^ gets latch level

SQLgt oradebug peek 0xampladdr 24

[2000C814 2000C82C) = 2000001F 0000007C 0000007E 00000006 pid^ gets latch level

983139983151983150983156 983155983153983148983152983148983157983155 983150983151983148983151983143 9831409831419831499831519831350128

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2982

LATCH FIXED TABLES

bull Before Oracle 11g all the latches were externalized to SQL inv$latch (x$ksllt)

bull In 11g parent (fixed SGA) latches are externalized inv$latch_parent (x$kslltr_parent)

bull In 11g child (allocated on startup) latches are externalized in

v$latch_children (x$kslltr_children) bull Oracle 11g v$latch (x$kslltr) merges these data Beware select

from x$kslltr acquires all parent latches and their children

bull Only part of latch structure is visible in x$ksllthellip fixed tables ndash kslltwhr kslltwhy ndash ldquowhererdquo and ldquowhyrdquo latch was acquired last time

ndash kslltwgt hellip - statistics

ndash kslltlvl hellip - attributes

ndash kslltwtt ndash latch wait time

RDTEX | Latch internals | RUOUG Seminar

29

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3082

CUMULATIVE V$LATCH STATISTIC COUNTERS

RDTEX | Latch internals | RUOUG Seminar

Statistics x$ksllt When and how it increments

GETS kslltwgt ldquo++rdquo after wait mode latch get

MISSES kslltwff ldquo++rdquo after wait get if it was missed

SLEEPS kslltwsl ldquo+number_of_sleepsrdquo during get

SPIN_GETS ksllthst0 ldquo++rdquo if get was missed but not slept

WAIT_TIME kslltwtt ldquo+wait_timerdquo after latch get

IMMEDIATE_GETS kslltngt ldquo++rdquo after nowait mode latch get Is notincremented on miss

IMMEDIATE_MISSES kslltnfa ldquo++rdquo if nowait mode get was missed

30

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3182

MANY STATISTICS WERE LOST IN 102

In 102 Oracle made obsolete many useful latch statistics

Latch size was decreased and the following fields always contain 0now in v$latch

ndash v$latchwaiters_woken

ndash v$latchwaits_holding_latch

ndash v$latchsleep1hellipsleep10

ndash and many undocumented statistics

This was mentioned in 102 Reference

RDTEX | Latch internals | RUOUG Seminar

31

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3282

X$KSLLD - LATCH DESCRIPTORS

bull Flags and attributes describing each latch are stored in kslldt arrayof kslld structures

bull [K]ernel [S]ervice [L]atch [L]ock [D]escriptor

bull Some but not all of kslld fields are externalized via x$kslld andv$latchname

bull Located in the PGA

RDTEX | Latch internals | RUOUG Seminar

32

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382

LATCH ATTRIBUTES

bull Each latch have at least the following attributes in kslldt

ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared

ndash PAR Is the latch Solitary or Parent for the family of child latches

ndash G2C Can two child latches be simultaneously requested in wait mode

using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92

ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above

ndash Level 0-14 Used to prevent latch deadlocks

ndash Class 0-7 Spin and wait class assigned to the latch 92 and above

RDTEX | Latch internals | RUOUG Seminar

33

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482

LATCH ATTRIBUTES BY ORACLE VERSION

RDTEX | Latch internals | RUOUG Seminar

11203 553 154 72 - 6 93

Oracle Number of latches PAR G2C LNG UFS SHARED

7340 53 14 2 3 - -

8063 80 21 7 3 - 38174 152 48 19 4 - 9

9208 242 79 37 - - 19

10202 385 114 55 - 4 47

10203 388 117 58 - 4 48

10204 394 117 59 - 4 50

11106 496 145 67 - 6 81

11107 502 145 67 - 6 83

11201 535 149 70 - 6 86

11202 551 153 72 - 6 91

34

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582

LATCH LEVELS

bull Every latch has a level in the range 0-14 to avoid deadlocks

bull Process can request a latch X in wait mode after obtaining latch Y if and

only if level X gt level Y

bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y

bull If these rules are broken the Oracle process raises

ORA-600[504] - trying to obtain another latch at incompatible level Note 281041

ORA-600[525] - trying to obtain another latch at the same level Note 1388881

ORA-600[526] - latch child order violation Note 1388891

RDTEX | Latch internals | RUOUG Seminar

35

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682

PARENT AND CHILD LATCHES

bull Latch can be a parent child or solitary latch

bull Both parent and child latches share the same latch name

bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss

bull Child latches are allocated during instance startup

bull The parent latch can be gotten independently

bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s

RDTEX | Latch internals | RUOUG Seminar

(latch info) wait_event=0 bits=10

holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl

Context saved from call 0

state=busy

36

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782

LATCH TREES

ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches

Direct SGA access program output for 9206 instance with too smallshared pool

RDTEX | Latch internals | RUOUG Seminar

ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1

waiter ospid 129 sid 72 pid 45

holding a154b7120 library cache (157) level=5 child=17 kglupc child

waiter ospid 18255 sid 65 pid 930

waiter ospid 6690 sid 554 pid 1654

waiter ospid 4685 sid 879 pid 1034

hellip

waiter ospid 29749 sid 180 pid 155

holding a154b7db8 library cache (157) level=5 child=4 kglupc child

waiter ospid 13104 sid 281 pid 220

waiter ospid 24089 sid 565 pid 636

waiter ospid 25002 sid 621 pid 1481

waiter ospid 16930 sid 1046 pid 783hellip

37

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882

LATCH CLASSES

bull Classes with different spin and wait policies

bull Up to 8 classes By default all the latches belong to class 0

bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2

bull Latch can be assigned to class by its number For example

ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000

ndash _LATCH_CLASSES=46

bull Will be discussed later

RDTEX | Latch internals | RUOUG Seminar

SQLgt select from x$ksllclass

INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4

---- ------ ----- -------- ------ ------ ------ ------ ------

0 20000 0 1 8000 8000 8000 8000 8000

hellip

6 100 12 1000 2000 3000 4000 4000 4000

38

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982

LATCH ACQUISITION IN NO-WAIT MODE

bull kslgetl(laddr0whywhere) ndash Exclusive latch get

bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep

ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches

ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches

bull IMMEDIATE_ statistics are updated

bull Used when many candidate latches are available to bypass the

latch level rule or as part of CBC chain examination

RDTEX | Latch internals | RUOUG Seminar

sqlplus nolog demo_03sql39

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082

LATCH WAITS

RDTEX | Latch internals | RUOUG Seminar 40

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182

WAITING FOR THE LATCH

RDTEX | Latch internals | RUOUG Seminar

983123 G A

C983120983125 1 C983120983125 2

983120983154983151983139983141983155983155 A

983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)

983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144

983116983137983156983139983144

983120983154983151983139983141983155983155 B

41

RDTEX | L h i l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282

LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION

bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries

again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of

times

bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep

bull A sleep corresponds to a wait for the latch free wait event

bull Initially a process sleeps for 001 s This time is doubled in every

subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip

RDTEX | Latch internals | RUOUG Seminar

Still did not found any versionwhich used this

May be it was lt= Oracle 6

42

RDTEX | L t h i t l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382

LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81

Latch gets with wait (kslgetl(laddress1hellip))

ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times

ndash Sleep on latch free event with exponential backoff

ndash Repeat

RDTEX | Latch internals | RUOUG Seminar

43

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482

8I LATCH SPIN CODE FLOW USING DTRACE

RDTEX | Latch internals | RUOUG Seminar

bull hellip Event 10046 trace

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2

983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29

983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270

sqlplus nolog demo_04_8sql 29

44

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582

POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF

bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec

RDTEX | Latch internals | RUOUG Seminar

12 ]2 )1[(

minus= +wait N

timeout

45

bull Most waits were for nothing ndash latch already was free

bull Latch utilization can not be high

bull Unnecessary spins provoke CPU thrashing

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682

92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE

sqlplus nolog demo_05sql 45

RDTEX | Latch internals | RUOUG Seminar

kslgetl(0x50006318 1)

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt kslges(0x50006318 ) -wait latch get

-gt skgslsgts(0x50006318 ) -spin latch get

-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT

-gt kskthbwt(0x0)

-gt kslwlmod() - set up Wait List

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt skgpwwait -sleep latch getsemop(11 17-10 1)

983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140

46

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782

RELIABLE LATCH WAITS

bull Hidden latch wait revolution that we missed Latch relies on waitposting

bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout

bull Exclusive latch spins 20000 cycles by default

bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock

bull Latches assigned to non-default class wait until timeout

bull Version 90 was transient and used mix of backoff and wait posting

This is why _LATCH_WAIT_POSTING parameter disappeared in9i

bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin

RDTEX | Latch internals | RUOUG Seminar

47

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882

IMAGINE THE TIME MICROSCOPE

| a a s | a

Reality One million zoomed time

1 us 1 sec

1 ms 17 min1 sec 115 days

Light speed (300000 kmsec) Sonic speed (300 msec)

CPU tick (2 GHz) 00005 sec

Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec

Max spin time for shared latch (~220005 ticks) ~10 sec

Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec

Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec

OS context switch time (10us-1ms) 10sec-17min

48

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982

LATCH WAITS UNDER TIME MICROSCOPE

Old 8i latch algorithmSpin for latch during 5 seconds

Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min

hellipSpin again for 5 seconds

Sleep again for 23 days (2 sec) hellip

New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)

If spin not succeeded go to sleep until post

According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)

| |

49

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082

_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES

bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and

_LATCH_CLASS_0 parameters

bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000

bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter

bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch

50

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182

TIMED LATCH WAIT POSTING

bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel

bull This can lead to instance hang ndash process will never be woken up

bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout

bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem

bull Generic AIX ports always use timed latch wait posting

bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux

51

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282

FINE GRAIN LATCH TUNING

bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting

bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7

bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo

microseconds using pollsys() (not semtimedop()) system call

ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times

Loop up to ldquoSpinsrdquo cycles

Yield CPU using yield() (or sched_yield())

ndash Sleep for ldquoSleepXrdquo usecs

ndash Then spin again hellip

52

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382

TRACES OF NONDEFAULT CLASS LATCH

983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152

991270

9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000

991270 E983158983141983150983156 10046 983156983154983137983139983141

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482

PERFORMANCE OF LATCH CLASSES

Row cache objects latch contention testcase on ODA (24 SMT)54

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582

IS THE _SPIN_COUNT REALLY STATIC

bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch

waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used

ldquocache buffers chainrdquo latch contention

bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass

(_LATCH_CLASS_X) By default ndash 20000

bull Shared latch spin can be tuned by _SPIN_COUNT value By

default ndash 2000 However this is not trivial

55

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682

SHARED LATCH BEHAVE LIKE ENQUEUE

bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80

bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests

bull This blocking achieved by special 0x40000000 bit in the latch

value This bit is a flag that some incompatible latch acquisitionis in progress

bull Queued processes will be woken up one by one on latch release

bull X mode latch gets effectively serialize the shared latch

bull See my blog for description of experiments

56

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782

TRACE OF SHARED LATCH WAIT

Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the

latch wait list and then spins again

983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141

983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156

983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983147983155983148983159983148983149983151983140()

983147983155983148983155983147983143983155(098316050009BAC )

983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140

983155983141983149983151983152(27)

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882

SHARED LATCH ACQUISITION

bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned

bull X mode shared latch get spins by default up to 4000 cycles

bull S mode does not spin at all (or spins in unknown way)

S mode get X mode get

Held in S mode Compatible 2_SPIN_COUNT

Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT

58

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982

LATCH RELEASE

bull Free the latch ndash kslfre(laddr)

bull Oracle process releases the latch nonatomically

bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process

bull This requires less bus invalidation and ensures propagation of latch

release to other local cachesbull Not fair policy Spinners on the local CPU board have the

preference

bull Then process posts the first process in the list of waiters

59

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082

LATCH CONTENTION

RDTEX | Latch internals | RUOUG Seminar60

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182

LATCH CONTENTION

bull Occurs when a latch is required by several processes at thesame time

bull Mostly due to bugs or abnormal application behavior

bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp

bull To diagnose the latch contention we need to investigate thelatch statistics [8]

61

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282

DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS

Latch requests arrival rate

Immediate gets efficiency

Latch sleeps ratio

Latch wait time per second

Latch spin efficiencymisses

getsspin∆

∆= _σ

misses

sleepsK

∆=

time

gets

∆=λ

gets

misses

∆= ρ

time

timewait W

∆=

_

Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics

62

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382

SAMPLED AVERAGES

By sampling of x$ksupr we can estimate

ndash Length of latch wait queue

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS

ndash Number of spinning processes in 10g

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS

Sampling of v$lathholder estimates latch utilization

Using DTrace we can measure

ndash Latch acquisition and holding times

ndash Actual spin count

w L

s N

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482

DERIVED LATCH STATISTICS

Latch utilization (PASTA)

Average holding time

Length of latch wait list

Recurrent sleeps ratio

Latch acquisition time

time

timeholdinglatchU

∆=asymp

__ ρ

Re_100

___

questsGet

TimeSnap MissGet Pct S

lowast==

λ

W L =

κ

κ σ 1minus+

)(1

W N T saq += minusλ

64

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 5: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 582

CONTEMPORARY LATCHES SINCE 92

RDTEX | Latch internals | RUOUG Seminar

5

I was wondering to find that

bull Exclusive latch spins 10 times morebull The exponential backoff was disappeared more then decade

ago in Oracle 92

bull All except one latch use wait posting now

bull Latch free wait may be infinite

bull Sometimes it is worth to tune the _SPIN_COUNT

bull This parameter is static for most latches

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 682

GREAT SOURCES OF LATCH INFO

ldquoOracle8i Internal Services for Waits Latches Locks and

Memoryrdquo by Steve Adams [1] 1999 ndash Founded new era of Oracle performance tuning

ndash However the Oracle latch technologies were evolved greatly duringthe decade

ldquoSystematic Latch Contention Troubleshootingrdquo approach byTanel Poder [2]

ndash Latchprofxsql script revolutionized the latch tuning

ldquoOracle Core Essential Internals for DBAs and Developersby Jonathan Lewis [3] 2011

RDTEX | Latch internals | RUOUG Seminar

6

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 782

REVIEW OF SERIALIZATION MECHANISMS IN ORACLE

bull Latches are simple low-level serialization mechanisms that coordinatemultiuser access to shared data structures objects and fileshellip

bull A mutual exclusion object (mutex) is a low-level mechanism thatprevents an object in memory from aging out or from being corrupted hellip

bull Internal locks are higher-level more complex mechanisms hellip

Oraclereg Database Concepts 112

bull KGX Mutexes appeared in latest Oracle versions inside Library Cache

RDTEX | Latch internals | RUOUG Seminar

A983139983139983141983155983155

A983139983153983157983145983155983145983156983145983151983150

983123983152983145983150983148983151983139983147 983138983137983155983141983140

983124983145983149983141983155983139983137983148983141

983116983145983142983141983139983161983139983148983141

983116983151983139983147983155

983123983141983158983141983154983137983148 983117983151983140983141983155

FIF983119

983118983151

gt 983117983145983148983148983145983155983141983139983151983150983140983155

D983161983150983137983149983145983139

983116983137983156983139983144983141983155

983124983161983152983141983155 983137983150983140 983117983151983140983141983155

983123I983122983119 (983155983152983145983150) + FIF983119

983129983141983155

983117983145983139983154983151983155983141983139983151983150983140983155

983123983156983137983156983145983139

983117983157983156983141983160983141983155

983119983152983141983154983137983156983145983151983150983155

983123I983122983119

983129983141983155

983123983157983138983117983145983139983154983151983155983141983139983151983150983140983155

D983161983150983137983149983145983139 7

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 882

HOW SPINLOCK WORKS

RDTEX | Latch internals | RUOUG Seminar

8

CPU3

Free FreeHolding by CPU 1

Spin polls the spinlock location

Get

Miss Sleep Spin count ( )∆

Holding time (S )

CPU2 Spin Holding the spinlock

Holding by CPU 2

CPU1 Holding the spinlock

The spinlock location

Sleep releases the CPU

bull Oracle latches

ndash Use atomic hardware instruction for Immediate get

ndash If missed process spins by polling latch location during Spin get

ndash Number of spin cycles is bounded by spin count

ndash In spin get not succeed the process acquiring the latch sleeps

hellip

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 982

CLASSIC SPINLOCKS

bull Wiki hellip spinlock hellip waits in a loop repeatedly checking until the lockbecomes available hellip

bull Introduced by Edsger Dijkstra [4] in 1965 Have been thoroughlyinvestigated since that time [5]

bull Many sophisticated spinlock realizations were proposed and evaluated

(TS TTS Delay MCS Anderson)

bull Two general types

ndash System spinlocks Kernel OS threads cannot wait Major metrics

atomic operations frequency Shared bus utilization

ndash User spinlocks Oracle latch and mutex Average lock holding time ~10 us It is more efficient to poll a lock rather than pre-empt the threaddoing 1 ms context switch Metrics CPU and elapsed times

RDTEX | Latch internals | RUOUG Seminar

9

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1082

THE NEED FOR SPIN

RDTEX | Latch internals | RUOUG Seminar

10

Time to complete CBC latch contention testcase vs number of threads

No spinning _spin_count=0

_spin_count=2000

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1182

SPINLOCK REALIZATIONS

RDTEX | Latch internals | RUOUG Seminar

Spinlock Pseudocode Problems

TS

pre-112 mutex

while(Test_and_Set(lock)) Bus saturation

TTS

Oracle latch

while(lock||Test_and_Set(lock)) Invalidation storms onrelease (ldquoopen doorrdquo

ldquothundering herdsrdquo)

Delay

Latch and 112

Mutex

Adjustable delay Slower under contention

Anderson MCSetc

Queues Widely used in JavaLinux kernel hellip

CPU and memoryoverhead preemptionissues

11

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1282

THE TOOLS

RDTEX | Latch internals | RUOUG Seminar12

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1382

DTRACE SOLARIS DYNAMIC TRACING FRAMEWORK

DTrace allows us to investigate how Oracle latches perform in real time ndash Create triggers on any event inside Solaris and function call inside Oracle

providermodulefunctionname

pid1910oraclekslgetlentry ndash Write trigger bodies ndash actions May read and change any address location ndash Can count the latch spins trace the latch waits perform experiments ndash Measure times and distributions up to microseconds precision

RDTEX | Latch internals | RUOUG Seminar

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313500

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1482

ORADEBUG INTERNAL ORACLE DEBUGGER

RDTEX | Latch internals | RUOUG Seminar

bull oradebug call Allows us invoke any internal Oracle function manually

SQLgt oradebug peek 200222A0 24

[200222A0 200222B8) = 00000016 00000001 000001D0 00000007

SQLgt oradebug call kslgetl 0x5001A020 1 2 3Function returned 1

bull oradebug peek Examines contents of any memory location

bull oradebug poke Modifies memory

bull oradebug watch Watch a region of memory

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1582

LATCH CONTENTION TESTCASES

RDTEX | Latch internals | RUOUG Seminar

15

create table latch_contention as select from dualhellipfor i in 11000000loopselect dummy into a from latch_contention

end loop

bull Exclusive and shared latches behave differentlybull Contention for shared cache buffer chains latch arises when manysessions scan the same block concurrently and _fastpin_enable=0

bull To induce the contention for exclusive ldquorow cache objectsrdquo latch protectingdc_users row cache just add several dynamic RLS policies

exec dbms_rlsadd_policy(systemlatch_contentionrls_1 hellip

exec dbms_rlsadd_policy(systemlatch_contentionrls_2 helliphellip

bull These dummy policies generate predicates 1=1bull Many other contention scenarios possible

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313509831351

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1682

ROW CACHE LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

16

Elapsed and CPU times vs number of threadsOracle Database Appliance (24 SMT)

Latch wait

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1782

HOW ORACLEREQUESTS THE LATCH

RDTEX | Latch internals | RUOUG Seminar17

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1882

DTRACE REVEALS LATCH INTERFACE ROUTINES

Oracle calls the following functions to acquire the latch

ndash kslgetl(laddr wait why where) - get exclusive latch ndash kslg2c(l1l2trcwhy where) - get two excl child latches

ndash kslgetsl (laddrwaitwhywheremode) - get shared latch

bull 11g - ksl_get_shared_latch (hellip)

ndash kslg2cs(l1l2modetrcwhy where)) - get two shr child latches

ndash kslgpl(laddrcommentwhywhere) - get parent and all childs

ndash kslfre(laddr) - free the latch

Oracle give us possibility to do the same by oradebug call

RDTEX | Latch internals | RUOUG Seminar

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313501

983125983155983141 32983085983138983145983156 983152983151983154983156983155

64983138983145983156 O983154983137983139983148983141 983139983151983154983154983157983152983156983155 983156983144983141 983137983154983143983157983149983141983150983156983155 983151983142 983151983154983137983140983141983138983157983143 983139983137983148983148 983145983150 10204 983137983150983140 983137983138983151983158983141

18

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1982

ldquoWHERErdquo AND ldquoWHYrdquo ORACLE GETS THE LATCH

To request the latch Oracle kernel routines need

bull laddres ndash address of latch in SGAbull wait ndash flag for no-wait (0) or wait (1) latch acquisition

bull where ndash integer code for location from where the latch is acquired

Oracle lists possible latch ldquowhererdquo values in x$ksllwindxbull why - context of why the latch is acquiring at this ldquowhererdquo It

contains DBA for CBC latches (Tanel Poder) SGA chunk addressfor shared pool latch session address etchellip ldquoWhyrdquo meaning for

some ldquowhererdquo can be found in x$ksllwksllwlbl

bull mode ndash requesting state for shared lathes

ndash 8 SHARED mode 16 EXCLUSIVE mode

RDTEX | Latch internals | RUOUG Seminar

19

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2082

LIST OF LATCH ldquoWHERErdquo LOCATIONS

X$KSLLW ndash List of latch [W]here Part of v$latch_misses

ndash ksllwnam ndash Kernel function name(ldquokghupr1rdquo ldquoksqdelrdquo etc) ndash ksllwlbl ndash Description of ldquowhyrdquo context

Join this code on indx to

ndash x$ksuprlatksulawhr (v$latchholder)ndash ldquowhererdquo value that wasused by latch holder

ndash x$kslltkslltwhr (v$latch) ndash ldquowhererdquo that was used by lastsuccessful latch acquisition

ndash x$ksuprksllawer (v$process) ndash ldquowhererdquo that was used byprocess is being waiting for latch

ndash x$kslwscindx (v$latch_misses) ndash latch sleep statistics by

ldquowhererdquo

RDTEX | Latch internals | RUOUG Seminar

20

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2182

V$LATCH_MISSES

bull Based on X$KSLWSC - No[W]ait and [S]leep [C]ount stats byContext Documentation is rather obscure

bull The following counters are incremented on each latch sleep

sleep_count (ldquowhererdquo the latch was held)

wtr_slp_count (ldquowhererdquo the latch get was requested)

bull nwfail_count is incremented only for session which SID is equalto _LATCH_MISS_STAT_SID parameter

bull longhold_count is incremented for shared latches only when

someone held the latch at this ldquowhererdquo for the entire duration ofsleep

RDTEX | Latch internals | RUOUG Seminar

21

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2282

LATCH IS HOLDING BY PROCESS NOT SESSION

bull

RDTEX | Latch internals | RUOUG Seminar

983123983156983154983157983139983156 983147983155983157983152983154 983163

991270

983123983156983154983157983139983156 983147983155983148983148983137983163983147983155983148983148983156 98314798315598314898314898313798314898313798315698313114983133

983165

991270983165

983120983154983151983139983141983155983155 983142983145983160983141983140 983137983154983154983137983161

983158$983152983154983151983139983141983155983155 983085gt 983160$983147983155983157983152983154

983123983156983154983157983139983156 983147983155983148983148983156983163

991270

983165

L983145983155983156 983151983142 983137983148983148 983148983137983156983139983144983141983155

983158$983148983137983156983139983144 983085gt983160$983147983155983148983148983156

Each process has an array of references to the latches it is holding

22

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2382

KSLLA IS HIDDEN IN V$PROCESS

bull The process latching info is the kslla structure embedded in theprocess state object

bull Some kslla fields were externalized to SQL

bull kslla contains array of latches holding by process at each level andinformation about the state of any current attempt to acquire a

latch Not externalized directly

bull When you query v$latchholder x$ksuprlat Oracle scansv$processx$ksupr in order to find all the processes that are

holding the latchesbull Tanel Poder uses high frequency sampling of v$latchholder to

systematically troubleshoot latch contention

RDTEX | Latch internals | RUOUG Seminar

23

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2482

X$KSUPRKSLLA FIELDS INSTRUMENT THE LATCH GET

bull ksllalaq ndash address of latch acquiring Populated during immediateget (and spin before 11g)

bull ksllawat - latch being waited for This is v$processlatchwait

bull ksllawhy ndash ldquowhyrdquo for the latch being waited for

bull ksllawere ndash ldquowhererdquo for the latch being waited for

bull ksllalow ndash bit array of levels of currently holding latches

bull ksllaspn - latch this process is spinning on v$processlatchspinNot populated since 81

bull ksllaps - inter-process post statistics

RDTEX | Latch internals | RUOUG Seminar

24

Generic ldquolatch freerdquo and 27 latch specific ldquolatch free rdquo wait events areassociated with sleep only Spin is invisible to Oracle Wait Interface

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2582

THE LATCH STRUCTURE ndash KSLLT

RDTEX | Latch internals | RUOUG Seminar

struct ksllt

ltLatchgt

ldquowhererdquo and ldquowhyrdquo

Level latch class other attributes

Statistics

Latch wait list header

hellip

25

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2682

LATCH SIZE BY VERSION

nix 32bit nix 64bit Windows 32bit

734 92 - 120

806 104 - 104817 104 144 104

901 200 160

920 196 240 200

1010 256 208

1020 - 11203 100 160 104

RDTEX | Latch internals | RUOUG Seminar

x$ksmfsv ndash list of all fixed SGA variables983123E983116EC983124 DI983123983124I983118C983124 983147983155983149983142983155983155983145983162

F983122983119983117 983160$983147983155983149983142983155983158

983127HE983122E 983147983155983149983142983155983156983161983152 = 983147983155983148983148983156

Latch structure was bigger in 101 due to additional latch statistics

26

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2782

ORACLE LATCH IS NOT JUST A SINGLE MEMORY LOCATION

bull Before 11g Value of first latch byte (word for shared latches) wasused to determine latch state

0x00 ndash latch is free

0xFF ndash exclusive latch is busy Was 0x01 in Oracle 7

0x010x02hellip - shared latch holding by 12 hellip processes simultaneously

0x20000000 | pid - shared latch holding exclusively

bull In 11g first latch word shows the pid of the latch holder

0x00 ndash latch is free0x12 ndash Oracle process with pid 18 holds the exclusive latch

RDTEX | Latch internals | RUOUG Seminar

27

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2882

BUSY ORACLE 11G LATCH IN MEMORY

bull 112 Exclusive latch (32-bit platform)

bull 112 Shared latch EXCLUSIVE mode

RDTEX | Latch internals | RUOUG Seminar

SQLgt oradebug peek 200222A0 24

[200222A0 200222B8) = 00000016 00000001 000001D0 00000007

pid^ gets latch level

SQLgt oradebug peek 0xampladdr 24

[2000C814 2000C82C) = 2000001F 0000007C 0000007E 00000006 pid^ gets latch level

983139983151983150983156 983155983153983148983152983148983157983155 983150983151983148983151983143 9831409831419831499831519831350128

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2982

LATCH FIXED TABLES

bull Before Oracle 11g all the latches were externalized to SQL inv$latch (x$ksllt)

bull In 11g parent (fixed SGA) latches are externalized inv$latch_parent (x$kslltr_parent)

bull In 11g child (allocated on startup) latches are externalized in

v$latch_children (x$kslltr_children) bull Oracle 11g v$latch (x$kslltr) merges these data Beware select

from x$kslltr acquires all parent latches and their children

bull Only part of latch structure is visible in x$ksllthellip fixed tables ndash kslltwhr kslltwhy ndash ldquowhererdquo and ldquowhyrdquo latch was acquired last time

ndash kslltwgt hellip - statistics

ndash kslltlvl hellip - attributes

ndash kslltwtt ndash latch wait time

RDTEX | Latch internals | RUOUG Seminar

29

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3082

CUMULATIVE V$LATCH STATISTIC COUNTERS

RDTEX | Latch internals | RUOUG Seminar

Statistics x$ksllt When and how it increments

GETS kslltwgt ldquo++rdquo after wait mode latch get

MISSES kslltwff ldquo++rdquo after wait get if it was missed

SLEEPS kslltwsl ldquo+number_of_sleepsrdquo during get

SPIN_GETS ksllthst0 ldquo++rdquo if get was missed but not slept

WAIT_TIME kslltwtt ldquo+wait_timerdquo after latch get

IMMEDIATE_GETS kslltngt ldquo++rdquo after nowait mode latch get Is notincremented on miss

IMMEDIATE_MISSES kslltnfa ldquo++rdquo if nowait mode get was missed

30

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3182

MANY STATISTICS WERE LOST IN 102

In 102 Oracle made obsolete many useful latch statistics

Latch size was decreased and the following fields always contain 0now in v$latch

ndash v$latchwaiters_woken

ndash v$latchwaits_holding_latch

ndash v$latchsleep1hellipsleep10

ndash and many undocumented statistics

This was mentioned in 102 Reference

RDTEX | Latch internals | RUOUG Seminar

31

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3282

X$KSLLD - LATCH DESCRIPTORS

bull Flags and attributes describing each latch are stored in kslldt arrayof kslld structures

bull [K]ernel [S]ervice [L]atch [L]ock [D]escriptor

bull Some but not all of kslld fields are externalized via x$kslld andv$latchname

bull Located in the PGA

RDTEX | Latch internals | RUOUG Seminar

32

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382

LATCH ATTRIBUTES

bull Each latch have at least the following attributes in kslldt

ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared

ndash PAR Is the latch Solitary or Parent for the family of child latches

ndash G2C Can two child latches be simultaneously requested in wait mode

using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92

ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above

ndash Level 0-14 Used to prevent latch deadlocks

ndash Class 0-7 Spin and wait class assigned to the latch 92 and above

RDTEX | Latch internals | RUOUG Seminar

33

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482

LATCH ATTRIBUTES BY ORACLE VERSION

RDTEX | Latch internals | RUOUG Seminar

11203 553 154 72 - 6 93

Oracle Number of latches PAR G2C LNG UFS SHARED

7340 53 14 2 3 - -

8063 80 21 7 3 - 38174 152 48 19 4 - 9

9208 242 79 37 - - 19

10202 385 114 55 - 4 47

10203 388 117 58 - 4 48

10204 394 117 59 - 4 50

11106 496 145 67 - 6 81

11107 502 145 67 - 6 83

11201 535 149 70 - 6 86

11202 551 153 72 - 6 91

34

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582

LATCH LEVELS

bull Every latch has a level in the range 0-14 to avoid deadlocks

bull Process can request a latch X in wait mode after obtaining latch Y if and

only if level X gt level Y

bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y

bull If these rules are broken the Oracle process raises

ORA-600[504] - trying to obtain another latch at incompatible level Note 281041

ORA-600[525] - trying to obtain another latch at the same level Note 1388881

ORA-600[526] - latch child order violation Note 1388891

RDTEX | Latch internals | RUOUG Seminar

35

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682

PARENT AND CHILD LATCHES

bull Latch can be a parent child or solitary latch

bull Both parent and child latches share the same latch name

bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss

bull Child latches are allocated during instance startup

bull The parent latch can be gotten independently

bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s

RDTEX | Latch internals | RUOUG Seminar

(latch info) wait_event=0 bits=10

holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl

Context saved from call 0

state=busy

36

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782

LATCH TREES

ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches

Direct SGA access program output for 9206 instance with too smallshared pool

RDTEX | Latch internals | RUOUG Seminar

ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1

waiter ospid 129 sid 72 pid 45

holding a154b7120 library cache (157) level=5 child=17 kglupc child

waiter ospid 18255 sid 65 pid 930

waiter ospid 6690 sid 554 pid 1654

waiter ospid 4685 sid 879 pid 1034

hellip

waiter ospid 29749 sid 180 pid 155

holding a154b7db8 library cache (157) level=5 child=4 kglupc child

waiter ospid 13104 sid 281 pid 220

waiter ospid 24089 sid 565 pid 636

waiter ospid 25002 sid 621 pid 1481

waiter ospid 16930 sid 1046 pid 783hellip

37

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882

LATCH CLASSES

bull Classes with different spin and wait policies

bull Up to 8 classes By default all the latches belong to class 0

bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2

bull Latch can be assigned to class by its number For example

ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000

ndash _LATCH_CLASSES=46

bull Will be discussed later

RDTEX | Latch internals | RUOUG Seminar

SQLgt select from x$ksllclass

INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4

---- ------ ----- -------- ------ ------ ------ ------ ------

0 20000 0 1 8000 8000 8000 8000 8000

hellip

6 100 12 1000 2000 3000 4000 4000 4000

38

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982

LATCH ACQUISITION IN NO-WAIT MODE

bull kslgetl(laddr0whywhere) ndash Exclusive latch get

bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep

ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches

ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches

bull IMMEDIATE_ statistics are updated

bull Used when many candidate latches are available to bypass the

latch level rule or as part of CBC chain examination

RDTEX | Latch internals | RUOUG Seminar

sqlplus nolog demo_03sql39

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082

LATCH WAITS

RDTEX | Latch internals | RUOUG Seminar 40

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182

WAITING FOR THE LATCH

RDTEX | Latch internals | RUOUG Seminar

983123 G A

C983120983125 1 C983120983125 2

983120983154983151983139983141983155983155 A

983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)

983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144

983116983137983156983139983144

983120983154983151983139983141983155983155 B

41

RDTEX | L h i l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282

LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION

bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries

again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of

times

bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep

bull A sleep corresponds to a wait for the latch free wait event

bull Initially a process sleeps for 001 s This time is doubled in every

subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip

RDTEX | Latch internals | RUOUG Seminar

Still did not found any versionwhich used this

May be it was lt= Oracle 6

42

RDTEX | L t h i t l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382

LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81

Latch gets with wait (kslgetl(laddress1hellip))

ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times

ndash Sleep on latch free event with exponential backoff

ndash Repeat

RDTEX | Latch internals | RUOUG Seminar

43

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482

8I LATCH SPIN CODE FLOW USING DTRACE

RDTEX | Latch internals | RUOUG Seminar

bull hellip Event 10046 trace

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2

983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29

983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270

sqlplus nolog demo_04_8sql 29

44

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582

POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF

bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec

RDTEX | Latch internals | RUOUG Seminar

12 ]2 )1[(

minus= +wait N

timeout

45

bull Most waits were for nothing ndash latch already was free

bull Latch utilization can not be high

bull Unnecessary spins provoke CPU thrashing

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682

92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE

sqlplus nolog demo_05sql 45

RDTEX | Latch internals | RUOUG Seminar

kslgetl(0x50006318 1)

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt kslges(0x50006318 ) -wait latch get

-gt skgslsgts(0x50006318 ) -spin latch get

-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT

-gt kskthbwt(0x0)

-gt kslwlmod() - set up Wait List

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt skgpwwait -sleep latch getsemop(11 17-10 1)

983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140

46

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782

RELIABLE LATCH WAITS

bull Hidden latch wait revolution that we missed Latch relies on waitposting

bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout

bull Exclusive latch spins 20000 cycles by default

bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock

bull Latches assigned to non-default class wait until timeout

bull Version 90 was transient and used mix of backoff and wait posting

This is why _LATCH_WAIT_POSTING parameter disappeared in9i

bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin

RDTEX | Latch internals | RUOUG Seminar

47

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882

IMAGINE THE TIME MICROSCOPE

| a a s | a

Reality One million zoomed time

1 us 1 sec

1 ms 17 min1 sec 115 days

Light speed (300000 kmsec) Sonic speed (300 msec)

CPU tick (2 GHz) 00005 sec

Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec

Max spin time for shared latch (~220005 ticks) ~10 sec

Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec

Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec

OS context switch time (10us-1ms) 10sec-17min

48

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982

LATCH WAITS UNDER TIME MICROSCOPE

Old 8i latch algorithmSpin for latch during 5 seconds

Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min

hellipSpin again for 5 seconds

Sleep again for 23 days (2 sec) hellip

New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)

If spin not succeeded go to sleep until post

According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)

| |

49

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082

_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES

bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and

_LATCH_CLASS_0 parameters

bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000

bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter

bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch

50

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182

TIMED LATCH WAIT POSTING

bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel

bull This can lead to instance hang ndash process will never be woken up

bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout

bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem

bull Generic AIX ports always use timed latch wait posting

bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux

51

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282

FINE GRAIN LATCH TUNING

bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting

bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7

bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo

microseconds using pollsys() (not semtimedop()) system call

ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times

Loop up to ldquoSpinsrdquo cycles

Yield CPU using yield() (or sched_yield())

ndash Sleep for ldquoSleepXrdquo usecs

ndash Then spin again hellip

52

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382

TRACES OF NONDEFAULT CLASS LATCH

983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152

991270

9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000

991270 E983158983141983150983156 10046 983156983154983137983139983141

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482

PERFORMANCE OF LATCH CLASSES

Row cache objects latch contention testcase on ODA (24 SMT)54

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582

IS THE _SPIN_COUNT REALLY STATIC

bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch

waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used

ldquocache buffers chainrdquo latch contention

bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass

(_LATCH_CLASS_X) By default ndash 20000

bull Shared latch spin can be tuned by _SPIN_COUNT value By

default ndash 2000 However this is not trivial

55

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682

SHARED LATCH BEHAVE LIKE ENQUEUE

bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80

bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests

bull This blocking achieved by special 0x40000000 bit in the latch

value This bit is a flag that some incompatible latch acquisitionis in progress

bull Queued processes will be woken up one by one on latch release

bull X mode latch gets effectively serialize the shared latch

bull See my blog for description of experiments

56

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782

TRACE OF SHARED LATCH WAIT

Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the

latch wait list and then spins again

983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141

983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156

983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983147983155983148983159983148983149983151983140()

983147983155983148983155983147983143983155(098316050009BAC )

983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140

983155983141983149983151983152(27)

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882

SHARED LATCH ACQUISITION

bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned

bull X mode shared latch get spins by default up to 4000 cycles

bull S mode does not spin at all (or spins in unknown way)

S mode get X mode get

Held in S mode Compatible 2_SPIN_COUNT

Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT

58

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982

LATCH RELEASE

bull Free the latch ndash kslfre(laddr)

bull Oracle process releases the latch nonatomically

bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process

bull This requires less bus invalidation and ensures propagation of latch

release to other local cachesbull Not fair policy Spinners on the local CPU board have the

preference

bull Then process posts the first process in the list of waiters

59

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082

LATCH CONTENTION

RDTEX | Latch internals | RUOUG Seminar60

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182

LATCH CONTENTION

bull Occurs when a latch is required by several processes at thesame time

bull Mostly due to bugs or abnormal application behavior

bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp

bull To diagnose the latch contention we need to investigate thelatch statistics [8]

61

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282

DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS

Latch requests arrival rate

Immediate gets efficiency

Latch sleeps ratio

Latch wait time per second

Latch spin efficiencymisses

getsspin∆

∆= _σ

misses

sleepsK

∆=

time

gets

∆=λ

gets

misses

∆= ρ

time

timewait W

∆=

_

Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics

62

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382

SAMPLED AVERAGES

By sampling of x$ksupr we can estimate

ndash Length of latch wait queue

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS

ndash Number of spinning processes in 10g

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS

Sampling of v$lathholder estimates latch utilization

Using DTrace we can measure

ndash Latch acquisition and holding times

ndash Actual spin count

w L

s N

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482

DERIVED LATCH STATISTICS

Latch utilization (PASTA)

Average holding time

Length of latch wait list

Recurrent sleeps ratio

Latch acquisition time

time

timeholdinglatchU

∆=asymp

__ ρ

Re_100

___

questsGet

TimeSnap MissGet Pct S

lowast==

λ

W L =

κ

κ σ 1minus+

)(1

W N T saq += minusλ

64

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 6: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 682

GREAT SOURCES OF LATCH INFO

ldquoOracle8i Internal Services for Waits Latches Locks and

Memoryrdquo by Steve Adams [1] 1999 ndash Founded new era of Oracle performance tuning

ndash However the Oracle latch technologies were evolved greatly duringthe decade

ldquoSystematic Latch Contention Troubleshootingrdquo approach byTanel Poder [2]

ndash Latchprofxsql script revolutionized the latch tuning

ldquoOracle Core Essential Internals for DBAs and Developersby Jonathan Lewis [3] 2011

RDTEX | Latch internals | RUOUG Seminar

6

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 782

REVIEW OF SERIALIZATION MECHANISMS IN ORACLE

bull Latches are simple low-level serialization mechanisms that coordinatemultiuser access to shared data structures objects and fileshellip

bull A mutual exclusion object (mutex) is a low-level mechanism thatprevents an object in memory from aging out or from being corrupted hellip

bull Internal locks are higher-level more complex mechanisms hellip

Oraclereg Database Concepts 112

bull KGX Mutexes appeared in latest Oracle versions inside Library Cache

RDTEX | Latch internals | RUOUG Seminar

A983139983139983141983155983155

A983139983153983157983145983155983145983156983145983151983150

983123983152983145983150983148983151983139983147 983138983137983155983141983140

983124983145983149983141983155983139983137983148983141

983116983145983142983141983139983161983139983148983141

983116983151983139983147983155

983123983141983158983141983154983137983148 983117983151983140983141983155

FIF983119

983118983151

gt 983117983145983148983148983145983155983141983139983151983150983140983155

D983161983150983137983149983145983139

983116983137983156983139983144983141983155

983124983161983152983141983155 983137983150983140 983117983151983140983141983155

983123I983122983119 (983155983152983145983150) + FIF983119

983129983141983155

983117983145983139983154983151983155983141983139983151983150983140983155

983123983156983137983156983145983139

983117983157983156983141983160983141983155

983119983152983141983154983137983156983145983151983150983155

983123I983122983119

983129983141983155

983123983157983138983117983145983139983154983151983155983141983139983151983150983140983155

D983161983150983137983149983145983139 7

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 882

HOW SPINLOCK WORKS

RDTEX | Latch internals | RUOUG Seminar

8

CPU3

Free FreeHolding by CPU 1

Spin polls the spinlock location

Get

Miss Sleep Spin count ( )∆

Holding time (S )

CPU2 Spin Holding the spinlock

Holding by CPU 2

CPU1 Holding the spinlock

The spinlock location

Sleep releases the CPU

bull Oracle latches

ndash Use atomic hardware instruction for Immediate get

ndash If missed process spins by polling latch location during Spin get

ndash Number of spin cycles is bounded by spin count

ndash In spin get not succeed the process acquiring the latch sleeps

hellip

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 982

CLASSIC SPINLOCKS

bull Wiki hellip spinlock hellip waits in a loop repeatedly checking until the lockbecomes available hellip

bull Introduced by Edsger Dijkstra [4] in 1965 Have been thoroughlyinvestigated since that time [5]

bull Many sophisticated spinlock realizations were proposed and evaluated

(TS TTS Delay MCS Anderson)

bull Two general types

ndash System spinlocks Kernel OS threads cannot wait Major metrics

atomic operations frequency Shared bus utilization

ndash User spinlocks Oracle latch and mutex Average lock holding time ~10 us It is more efficient to poll a lock rather than pre-empt the threaddoing 1 ms context switch Metrics CPU and elapsed times

RDTEX | Latch internals | RUOUG Seminar

9

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1082

THE NEED FOR SPIN

RDTEX | Latch internals | RUOUG Seminar

10

Time to complete CBC latch contention testcase vs number of threads

No spinning _spin_count=0

_spin_count=2000

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1182

SPINLOCK REALIZATIONS

RDTEX | Latch internals | RUOUG Seminar

Spinlock Pseudocode Problems

TS

pre-112 mutex

while(Test_and_Set(lock)) Bus saturation

TTS

Oracle latch

while(lock||Test_and_Set(lock)) Invalidation storms onrelease (ldquoopen doorrdquo

ldquothundering herdsrdquo)

Delay

Latch and 112

Mutex

Adjustable delay Slower under contention

Anderson MCSetc

Queues Widely used in JavaLinux kernel hellip

CPU and memoryoverhead preemptionissues

11

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1282

THE TOOLS

RDTEX | Latch internals | RUOUG Seminar12

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1382

DTRACE SOLARIS DYNAMIC TRACING FRAMEWORK

DTrace allows us to investigate how Oracle latches perform in real time ndash Create triggers on any event inside Solaris and function call inside Oracle

providermodulefunctionname

pid1910oraclekslgetlentry ndash Write trigger bodies ndash actions May read and change any address location ndash Can count the latch spins trace the latch waits perform experiments ndash Measure times and distributions up to microseconds precision

RDTEX | Latch internals | RUOUG Seminar

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313500

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1482

ORADEBUG INTERNAL ORACLE DEBUGGER

RDTEX | Latch internals | RUOUG Seminar

bull oradebug call Allows us invoke any internal Oracle function manually

SQLgt oradebug peek 200222A0 24

[200222A0 200222B8) = 00000016 00000001 000001D0 00000007

SQLgt oradebug call kslgetl 0x5001A020 1 2 3Function returned 1

bull oradebug peek Examines contents of any memory location

bull oradebug poke Modifies memory

bull oradebug watch Watch a region of memory

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1582

LATCH CONTENTION TESTCASES

RDTEX | Latch internals | RUOUG Seminar

15

create table latch_contention as select from dualhellipfor i in 11000000loopselect dummy into a from latch_contention

end loop

bull Exclusive and shared latches behave differentlybull Contention for shared cache buffer chains latch arises when manysessions scan the same block concurrently and _fastpin_enable=0

bull To induce the contention for exclusive ldquorow cache objectsrdquo latch protectingdc_users row cache just add several dynamic RLS policies

exec dbms_rlsadd_policy(systemlatch_contentionrls_1 hellip

exec dbms_rlsadd_policy(systemlatch_contentionrls_2 helliphellip

bull These dummy policies generate predicates 1=1bull Many other contention scenarios possible

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313509831351

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1682

ROW CACHE LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

16

Elapsed and CPU times vs number of threadsOracle Database Appliance (24 SMT)

Latch wait

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1782

HOW ORACLEREQUESTS THE LATCH

RDTEX | Latch internals | RUOUG Seminar17

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1882

DTRACE REVEALS LATCH INTERFACE ROUTINES

Oracle calls the following functions to acquire the latch

ndash kslgetl(laddr wait why where) - get exclusive latch ndash kslg2c(l1l2trcwhy where) - get two excl child latches

ndash kslgetsl (laddrwaitwhywheremode) - get shared latch

bull 11g - ksl_get_shared_latch (hellip)

ndash kslg2cs(l1l2modetrcwhy where)) - get two shr child latches

ndash kslgpl(laddrcommentwhywhere) - get parent and all childs

ndash kslfre(laddr) - free the latch

Oracle give us possibility to do the same by oradebug call

RDTEX | Latch internals | RUOUG Seminar

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313501

983125983155983141 32983085983138983145983156 983152983151983154983156983155

64983138983145983156 O983154983137983139983148983141 983139983151983154983154983157983152983156983155 983156983144983141 983137983154983143983157983149983141983150983156983155 983151983142 983151983154983137983140983141983138983157983143 983139983137983148983148 983145983150 10204 983137983150983140 983137983138983151983158983141

18

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1982

ldquoWHERErdquo AND ldquoWHYrdquo ORACLE GETS THE LATCH

To request the latch Oracle kernel routines need

bull laddres ndash address of latch in SGAbull wait ndash flag for no-wait (0) or wait (1) latch acquisition

bull where ndash integer code for location from where the latch is acquired

Oracle lists possible latch ldquowhererdquo values in x$ksllwindxbull why - context of why the latch is acquiring at this ldquowhererdquo It

contains DBA for CBC latches (Tanel Poder) SGA chunk addressfor shared pool latch session address etchellip ldquoWhyrdquo meaning for

some ldquowhererdquo can be found in x$ksllwksllwlbl

bull mode ndash requesting state for shared lathes

ndash 8 SHARED mode 16 EXCLUSIVE mode

RDTEX | Latch internals | RUOUG Seminar

19

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2082

LIST OF LATCH ldquoWHERErdquo LOCATIONS

X$KSLLW ndash List of latch [W]here Part of v$latch_misses

ndash ksllwnam ndash Kernel function name(ldquokghupr1rdquo ldquoksqdelrdquo etc) ndash ksllwlbl ndash Description of ldquowhyrdquo context

Join this code on indx to

ndash x$ksuprlatksulawhr (v$latchholder)ndash ldquowhererdquo value that wasused by latch holder

ndash x$kslltkslltwhr (v$latch) ndash ldquowhererdquo that was used by lastsuccessful latch acquisition

ndash x$ksuprksllawer (v$process) ndash ldquowhererdquo that was used byprocess is being waiting for latch

ndash x$kslwscindx (v$latch_misses) ndash latch sleep statistics by

ldquowhererdquo

RDTEX | Latch internals | RUOUG Seminar

20

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2182

V$LATCH_MISSES

bull Based on X$KSLWSC - No[W]ait and [S]leep [C]ount stats byContext Documentation is rather obscure

bull The following counters are incremented on each latch sleep

sleep_count (ldquowhererdquo the latch was held)

wtr_slp_count (ldquowhererdquo the latch get was requested)

bull nwfail_count is incremented only for session which SID is equalto _LATCH_MISS_STAT_SID parameter

bull longhold_count is incremented for shared latches only when

someone held the latch at this ldquowhererdquo for the entire duration ofsleep

RDTEX | Latch internals | RUOUG Seminar

21

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2282

LATCH IS HOLDING BY PROCESS NOT SESSION

bull

RDTEX | Latch internals | RUOUG Seminar

983123983156983154983157983139983156 983147983155983157983152983154 983163

991270

983123983156983154983157983139983156 983147983155983148983148983137983163983147983155983148983148983156 98314798315598314898314898313798314898313798315698313114983133

983165

991270983165

983120983154983151983139983141983155983155 983142983145983160983141983140 983137983154983154983137983161

983158$983152983154983151983139983141983155983155 983085gt 983160$983147983155983157983152983154

983123983156983154983157983139983156 983147983155983148983148983156983163

991270

983165

L983145983155983156 983151983142 983137983148983148 983148983137983156983139983144983141983155

983158$983148983137983156983139983144 983085gt983160$983147983155983148983148983156

Each process has an array of references to the latches it is holding

22

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2382

KSLLA IS HIDDEN IN V$PROCESS

bull The process latching info is the kslla structure embedded in theprocess state object

bull Some kslla fields were externalized to SQL

bull kslla contains array of latches holding by process at each level andinformation about the state of any current attempt to acquire a

latch Not externalized directly

bull When you query v$latchholder x$ksuprlat Oracle scansv$processx$ksupr in order to find all the processes that are

holding the latchesbull Tanel Poder uses high frequency sampling of v$latchholder to

systematically troubleshoot latch contention

RDTEX | Latch internals | RUOUG Seminar

23

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2482

X$KSUPRKSLLA FIELDS INSTRUMENT THE LATCH GET

bull ksllalaq ndash address of latch acquiring Populated during immediateget (and spin before 11g)

bull ksllawat - latch being waited for This is v$processlatchwait

bull ksllawhy ndash ldquowhyrdquo for the latch being waited for

bull ksllawere ndash ldquowhererdquo for the latch being waited for

bull ksllalow ndash bit array of levels of currently holding latches

bull ksllaspn - latch this process is spinning on v$processlatchspinNot populated since 81

bull ksllaps - inter-process post statistics

RDTEX | Latch internals | RUOUG Seminar

24

Generic ldquolatch freerdquo and 27 latch specific ldquolatch free rdquo wait events areassociated with sleep only Spin is invisible to Oracle Wait Interface

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2582

THE LATCH STRUCTURE ndash KSLLT

RDTEX | Latch internals | RUOUG Seminar

struct ksllt

ltLatchgt

ldquowhererdquo and ldquowhyrdquo

Level latch class other attributes

Statistics

Latch wait list header

hellip

25

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2682

LATCH SIZE BY VERSION

nix 32bit nix 64bit Windows 32bit

734 92 - 120

806 104 - 104817 104 144 104

901 200 160

920 196 240 200

1010 256 208

1020 - 11203 100 160 104

RDTEX | Latch internals | RUOUG Seminar

x$ksmfsv ndash list of all fixed SGA variables983123E983116EC983124 DI983123983124I983118C983124 983147983155983149983142983155983155983145983162

F983122983119983117 983160$983147983155983149983142983155983158

983127HE983122E 983147983155983149983142983155983156983161983152 = 983147983155983148983148983156

Latch structure was bigger in 101 due to additional latch statistics

26

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2782

ORACLE LATCH IS NOT JUST A SINGLE MEMORY LOCATION

bull Before 11g Value of first latch byte (word for shared latches) wasused to determine latch state

0x00 ndash latch is free

0xFF ndash exclusive latch is busy Was 0x01 in Oracle 7

0x010x02hellip - shared latch holding by 12 hellip processes simultaneously

0x20000000 | pid - shared latch holding exclusively

bull In 11g first latch word shows the pid of the latch holder

0x00 ndash latch is free0x12 ndash Oracle process with pid 18 holds the exclusive latch

RDTEX | Latch internals | RUOUG Seminar

27

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2882

BUSY ORACLE 11G LATCH IN MEMORY

bull 112 Exclusive latch (32-bit platform)

bull 112 Shared latch EXCLUSIVE mode

RDTEX | Latch internals | RUOUG Seminar

SQLgt oradebug peek 200222A0 24

[200222A0 200222B8) = 00000016 00000001 000001D0 00000007

pid^ gets latch level

SQLgt oradebug peek 0xampladdr 24

[2000C814 2000C82C) = 2000001F 0000007C 0000007E 00000006 pid^ gets latch level

983139983151983150983156 983155983153983148983152983148983157983155 983150983151983148983151983143 9831409831419831499831519831350128

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2982

LATCH FIXED TABLES

bull Before Oracle 11g all the latches were externalized to SQL inv$latch (x$ksllt)

bull In 11g parent (fixed SGA) latches are externalized inv$latch_parent (x$kslltr_parent)

bull In 11g child (allocated on startup) latches are externalized in

v$latch_children (x$kslltr_children) bull Oracle 11g v$latch (x$kslltr) merges these data Beware select

from x$kslltr acquires all parent latches and their children

bull Only part of latch structure is visible in x$ksllthellip fixed tables ndash kslltwhr kslltwhy ndash ldquowhererdquo and ldquowhyrdquo latch was acquired last time

ndash kslltwgt hellip - statistics

ndash kslltlvl hellip - attributes

ndash kslltwtt ndash latch wait time

RDTEX | Latch internals | RUOUG Seminar

29

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3082

CUMULATIVE V$LATCH STATISTIC COUNTERS

RDTEX | Latch internals | RUOUG Seminar

Statistics x$ksllt When and how it increments

GETS kslltwgt ldquo++rdquo after wait mode latch get

MISSES kslltwff ldquo++rdquo after wait get if it was missed

SLEEPS kslltwsl ldquo+number_of_sleepsrdquo during get

SPIN_GETS ksllthst0 ldquo++rdquo if get was missed but not slept

WAIT_TIME kslltwtt ldquo+wait_timerdquo after latch get

IMMEDIATE_GETS kslltngt ldquo++rdquo after nowait mode latch get Is notincremented on miss

IMMEDIATE_MISSES kslltnfa ldquo++rdquo if nowait mode get was missed

30

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3182

MANY STATISTICS WERE LOST IN 102

In 102 Oracle made obsolete many useful latch statistics

Latch size was decreased and the following fields always contain 0now in v$latch

ndash v$latchwaiters_woken

ndash v$latchwaits_holding_latch

ndash v$latchsleep1hellipsleep10

ndash and many undocumented statistics

This was mentioned in 102 Reference

RDTEX | Latch internals | RUOUG Seminar

31

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3282

X$KSLLD - LATCH DESCRIPTORS

bull Flags and attributes describing each latch are stored in kslldt arrayof kslld structures

bull [K]ernel [S]ervice [L]atch [L]ock [D]escriptor

bull Some but not all of kslld fields are externalized via x$kslld andv$latchname

bull Located in the PGA

RDTEX | Latch internals | RUOUG Seminar

32

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382

LATCH ATTRIBUTES

bull Each latch have at least the following attributes in kslldt

ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared

ndash PAR Is the latch Solitary or Parent for the family of child latches

ndash G2C Can two child latches be simultaneously requested in wait mode

using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92

ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above

ndash Level 0-14 Used to prevent latch deadlocks

ndash Class 0-7 Spin and wait class assigned to the latch 92 and above

RDTEX | Latch internals | RUOUG Seminar

33

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482

LATCH ATTRIBUTES BY ORACLE VERSION

RDTEX | Latch internals | RUOUG Seminar

11203 553 154 72 - 6 93

Oracle Number of latches PAR G2C LNG UFS SHARED

7340 53 14 2 3 - -

8063 80 21 7 3 - 38174 152 48 19 4 - 9

9208 242 79 37 - - 19

10202 385 114 55 - 4 47

10203 388 117 58 - 4 48

10204 394 117 59 - 4 50

11106 496 145 67 - 6 81

11107 502 145 67 - 6 83

11201 535 149 70 - 6 86

11202 551 153 72 - 6 91

34

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582

LATCH LEVELS

bull Every latch has a level in the range 0-14 to avoid deadlocks

bull Process can request a latch X in wait mode after obtaining latch Y if and

only if level X gt level Y

bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y

bull If these rules are broken the Oracle process raises

ORA-600[504] - trying to obtain another latch at incompatible level Note 281041

ORA-600[525] - trying to obtain another latch at the same level Note 1388881

ORA-600[526] - latch child order violation Note 1388891

RDTEX | Latch internals | RUOUG Seminar

35

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682

PARENT AND CHILD LATCHES

bull Latch can be a parent child or solitary latch

bull Both parent and child latches share the same latch name

bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss

bull Child latches are allocated during instance startup

bull The parent latch can be gotten independently

bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s

RDTEX | Latch internals | RUOUG Seminar

(latch info) wait_event=0 bits=10

holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl

Context saved from call 0

state=busy

36

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782

LATCH TREES

ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches

Direct SGA access program output for 9206 instance with too smallshared pool

RDTEX | Latch internals | RUOUG Seminar

ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1

waiter ospid 129 sid 72 pid 45

holding a154b7120 library cache (157) level=5 child=17 kglupc child

waiter ospid 18255 sid 65 pid 930

waiter ospid 6690 sid 554 pid 1654

waiter ospid 4685 sid 879 pid 1034

hellip

waiter ospid 29749 sid 180 pid 155

holding a154b7db8 library cache (157) level=5 child=4 kglupc child

waiter ospid 13104 sid 281 pid 220

waiter ospid 24089 sid 565 pid 636

waiter ospid 25002 sid 621 pid 1481

waiter ospid 16930 sid 1046 pid 783hellip

37

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882

LATCH CLASSES

bull Classes with different spin and wait policies

bull Up to 8 classes By default all the latches belong to class 0

bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2

bull Latch can be assigned to class by its number For example

ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000

ndash _LATCH_CLASSES=46

bull Will be discussed later

RDTEX | Latch internals | RUOUG Seminar

SQLgt select from x$ksllclass

INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4

---- ------ ----- -------- ------ ------ ------ ------ ------

0 20000 0 1 8000 8000 8000 8000 8000

hellip

6 100 12 1000 2000 3000 4000 4000 4000

38

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982

LATCH ACQUISITION IN NO-WAIT MODE

bull kslgetl(laddr0whywhere) ndash Exclusive latch get

bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep

ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches

ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches

bull IMMEDIATE_ statistics are updated

bull Used when many candidate latches are available to bypass the

latch level rule or as part of CBC chain examination

RDTEX | Latch internals | RUOUG Seminar

sqlplus nolog demo_03sql39

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082

LATCH WAITS

RDTEX | Latch internals | RUOUG Seminar 40

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182

WAITING FOR THE LATCH

RDTEX | Latch internals | RUOUG Seminar

983123 G A

C983120983125 1 C983120983125 2

983120983154983151983139983141983155983155 A

983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)

983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144

983116983137983156983139983144

983120983154983151983139983141983155983155 B

41

RDTEX | L h i l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282

LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION

bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries

again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of

times

bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep

bull A sleep corresponds to a wait for the latch free wait event

bull Initially a process sleeps for 001 s This time is doubled in every

subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip

RDTEX | Latch internals | RUOUG Seminar

Still did not found any versionwhich used this

May be it was lt= Oracle 6

42

RDTEX | L t h i t l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382

LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81

Latch gets with wait (kslgetl(laddress1hellip))

ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times

ndash Sleep on latch free event with exponential backoff

ndash Repeat

RDTEX | Latch internals | RUOUG Seminar

43

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482

8I LATCH SPIN CODE FLOW USING DTRACE

RDTEX | Latch internals | RUOUG Seminar

bull hellip Event 10046 trace

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2

983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29

983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270

sqlplus nolog demo_04_8sql 29

44

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582

POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF

bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec

RDTEX | Latch internals | RUOUG Seminar

12 ]2 )1[(

minus= +wait N

timeout

45

bull Most waits were for nothing ndash latch already was free

bull Latch utilization can not be high

bull Unnecessary spins provoke CPU thrashing

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682

92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE

sqlplus nolog demo_05sql 45

RDTEX | Latch internals | RUOUG Seminar

kslgetl(0x50006318 1)

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt kslges(0x50006318 ) -wait latch get

-gt skgslsgts(0x50006318 ) -spin latch get

-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT

-gt kskthbwt(0x0)

-gt kslwlmod() - set up Wait List

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt skgpwwait -sleep latch getsemop(11 17-10 1)

983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140

46

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782

RELIABLE LATCH WAITS

bull Hidden latch wait revolution that we missed Latch relies on waitposting

bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout

bull Exclusive latch spins 20000 cycles by default

bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock

bull Latches assigned to non-default class wait until timeout

bull Version 90 was transient and used mix of backoff and wait posting

This is why _LATCH_WAIT_POSTING parameter disappeared in9i

bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin

RDTEX | Latch internals | RUOUG Seminar

47

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882

IMAGINE THE TIME MICROSCOPE

| a a s | a

Reality One million zoomed time

1 us 1 sec

1 ms 17 min1 sec 115 days

Light speed (300000 kmsec) Sonic speed (300 msec)

CPU tick (2 GHz) 00005 sec

Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec

Max spin time for shared latch (~220005 ticks) ~10 sec

Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec

Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec

OS context switch time (10us-1ms) 10sec-17min

48

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982

LATCH WAITS UNDER TIME MICROSCOPE

Old 8i latch algorithmSpin for latch during 5 seconds

Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min

hellipSpin again for 5 seconds

Sleep again for 23 days (2 sec) hellip

New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)

If spin not succeeded go to sleep until post

According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)

| |

49

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082

_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES

bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and

_LATCH_CLASS_0 parameters

bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000

bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter

bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch

50

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182

TIMED LATCH WAIT POSTING

bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel

bull This can lead to instance hang ndash process will never be woken up

bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout

bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem

bull Generic AIX ports always use timed latch wait posting

bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux

51

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282

FINE GRAIN LATCH TUNING

bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting

bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7

bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo

microseconds using pollsys() (not semtimedop()) system call

ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times

Loop up to ldquoSpinsrdquo cycles

Yield CPU using yield() (or sched_yield())

ndash Sleep for ldquoSleepXrdquo usecs

ndash Then spin again hellip

52

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382

TRACES OF NONDEFAULT CLASS LATCH

983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152

991270

9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000

991270 E983158983141983150983156 10046 983156983154983137983139983141

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482

PERFORMANCE OF LATCH CLASSES

Row cache objects latch contention testcase on ODA (24 SMT)54

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582

IS THE _SPIN_COUNT REALLY STATIC

bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch

waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used

ldquocache buffers chainrdquo latch contention

bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass

(_LATCH_CLASS_X) By default ndash 20000

bull Shared latch spin can be tuned by _SPIN_COUNT value By

default ndash 2000 However this is not trivial

55

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682

SHARED LATCH BEHAVE LIKE ENQUEUE

bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80

bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests

bull This blocking achieved by special 0x40000000 bit in the latch

value This bit is a flag that some incompatible latch acquisitionis in progress

bull Queued processes will be woken up one by one on latch release

bull X mode latch gets effectively serialize the shared latch

bull See my blog for description of experiments

56

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782

TRACE OF SHARED LATCH WAIT

Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the

latch wait list and then spins again

983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141

983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156

983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983147983155983148983159983148983149983151983140()

983147983155983148983155983147983143983155(098316050009BAC )

983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140

983155983141983149983151983152(27)

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882

SHARED LATCH ACQUISITION

bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned

bull X mode shared latch get spins by default up to 4000 cycles

bull S mode does not spin at all (or spins in unknown way)

S mode get X mode get

Held in S mode Compatible 2_SPIN_COUNT

Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT

58

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982

LATCH RELEASE

bull Free the latch ndash kslfre(laddr)

bull Oracle process releases the latch nonatomically

bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process

bull This requires less bus invalidation and ensures propagation of latch

release to other local cachesbull Not fair policy Spinners on the local CPU board have the

preference

bull Then process posts the first process in the list of waiters

59

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082

LATCH CONTENTION

RDTEX | Latch internals | RUOUG Seminar60

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182

LATCH CONTENTION

bull Occurs when a latch is required by several processes at thesame time

bull Mostly due to bugs or abnormal application behavior

bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp

bull To diagnose the latch contention we need to investigate thelatch statistics [8]

61

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282

DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS

Latch requests arrival rate

Immediate gets efficiency

Latch sleeps ratio

Latch wait time per second

Latch spin efficiencymisses

getsspin∆

∆= _σ

misses

sleepsK

∆=

time

gets

∆=λ

gets

misses

∆= ρ

time

timewait W

∆=

_

Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics

62

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382

SAMPLED AVERAGES

By sampling of x$ksupr we can estimate

ndash Length of latch wait queue

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS

ndash Number of spinning processes in 10g

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS

Sampling of v$lathholder estimates latch utilization

Using DTrace we can measure

ndash Latch acquisition and holding times

ndash Actual spin count

w L

s N

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482

DERIVED LATCH STATISTICS

Latch utilization (PASTA)

Average holding time

Length of latch wait list

Recurrent sleeps ratio

Latch acquisition time

time

timeholdinglatchU

∆=asymp

__ ρ

Re_100

___

questsGet

TimeSnap MissGet Pct S

lowast==

λ

W L =

κ

κ σ 1minus+

)(1

W N T saq += minusλ

64

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 7: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 782

REVIEW OF SERIALIZATION MECHANISMS IN ORACLE

bull Latches are simple low-level serialization mechanisms that coordinatemultiuser access to shared data structures objects and fileshellip

bull A mutual exclusion object (mutex) is a low-level mechanism thatprevents an object in memory from aging out or from being corrupted hellip

bull Internal locks are higher-level more complex mechanisms hellip

Oraclereg Database Concepts 112

bull KGX Mutexes appeared in latest Oracle versions inside Library Cache

RDTEX | Latch internals | RUOUG Seminar

A983139983139983141983155983155

A983139983153983157983145983155983145983156983145983151983150

983123983152983145983150983148983151983139983147 983138983137983155983141983140

983124983145983149983141983155983139983137983148983141

983116983145983142983141983139983161983139983148983141

983116983151983139983147983155

983123983141983158983141983154983137983148 983117983151983140983141983155

FIF983119

983118983151

gt 983117983145983148983148983145983155983141983139983151983150983140983155

D983161983150983137983149983145983139

983116983137983156983139983144983141983155

983124983161983152983141983155 983137983150983140 983117983151983140983141983155

983123I983122983119 (983155983152983145983150) + FIF983119

983129983141983155

983117983145983139983154983151983155983141983139983151983150983140983155

983123983156983137983156983145983139

983117983157983156983141983160983141983155

983119983152983141983154983137983156983145983151983150983155

983123I983122983119

983129983141983155

983123983157983138983117983145983139983154983151983155983141983139983151983150983140983155

D983161983150983137983149983145983139 7

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 882

HOW SPINLOCK WORKS

RDTEX | Latch internals | RUOUG Seminar

8

CPU3

Free FreeHolding by CPU 1

Spin polls the spinlock location

Get

Miss Sleep Spin count ( )∆

Holding time (S )

CPU2 Spin Holding the spinlock

Holding by CPU 2

CPU1 Holding the spinlock

The spinlock location

Sleep releases the CPU

bull Oracle latches

ndash Use atomic hardware instruction for Immediate get

ndash If missed process spins by polling latch location during Spin get

ndash Number of spin cycles is bounded by spin count

ndash In spin get not succeed the process acquiring the latch sleeps

hellip

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 982

CLASSIC SPINLOCKS

bull Wiki hellip spinlock hellip waits in a loop repeatedly checking until the lockbecomes available hellip

bull Introduced by Edsger Dijkstra [4] in 1965 Have been thoroughlyinvestigated since that time [5]

bull Many sophisticated spinlock realizations were proposed and evaluated

(TS TTS Delay MCS Anderson)

bull Two general types

ndash System spinlocks Kernel OS threads cannot wait Major metrics

atomic operations frequency Shared bus utilization

ndash User spinlocks Oracle latch and mutex Average lock holding time ~10 us It is more efficient to poll a lock rather than pre-empt the threaddoing 1 ms context switch Metrics CPU and elapsed times

RDTEX | Latch internals | RUOUG Seminar

9

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1082

THE NEED FOR SPIN

RDTEX | Latch internals | RUOUG Seminar

10

Time to complete CBC latch contention testcase vs number of threads

No spinning _spin_count=0

_spin_count=2000

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1182

SPINLOCK REALIZATIONS

RDTEX | Latch internals | RUOUG Seminar

Spinlock Pseudocode Problems

TS

pre-112 mutex

while(Test_and_Set(lock)) Bus saturation

TTS

Oracle latch

while(lock||Test_and_Set(lock)) Invalidation storms onrelease (ldquoopen doorrdquo

ldquothundering herdsrdquo)

Delay

Latch and 112

Mutex

Adjustable delay Slower under contention

Anderson MCSetc

Queues Widely used in JavaLinux kernel hellip

CPU and memoryoverhead preemptionissues

11

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1282

THE TOOLS

RDTEX | Latch internals | RUOUG Seminar12

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1382

DTRACE SOLARIS DYNAMIC TRACING FRAMEWORK

DTrace allows us to investigate how Oracle latches perform in real time ndash Create triggers on any event inside Solaris and function call inside Oracle

providermodulefunctionname

pid1910oraclekslgetlentry ndash Write trigger bodies ndash actions May read and change any address location ndash Can count the latch spins trace the latch waits perform experiments ndash Measure times and distributions up to microseconds precision

RDTEX | Latch internals | RUOUG Seminar

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313500

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1482

ORADEBUG INTERNAL ORACLE DEBUGGER

RDTEX | Latch internals | RUOUG Seminar

bull oradebug call Allows us invoke any internal Oracle function manually

SQLgt oradebug peek 200222A0 24

[200222A0 200222B8) = 00000016 00000001 000001D0 00000007

SQLgt oradebug call kslgetl 0x5001A020 1 2 3Function returned 1

bull oradebug peek Examines contents of any memory location

bull oradebug poke Modifies memory

bull oradebug watch Watch a region of memory

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1582

LATCH CONTENTION TESTCASES

RDTEX | Latch internals | RUOUG Seminar

15

create table latch_contention as select from dualhellipfor i in 11000000loopselect dummy into a from latch_contention

end loop

bull Exclusive and shared latches behave differentlybull Contention for shared cache buffer chains latch arises when manysessions scan the same block concurrently and _fastpin_enable=0

bull To induce the contention for exclusive ldquorow cache objectsrdquo latch protectingdc_users row cache just add several dynamic RLS policies

exec dbms_rlsadd_policy(systemlatch_contentionrls_1 hellip

exec dbms_rlsadd_policy(systemlatch_contentionrls_2 helliphellip

bull These dummy policies generate predicates 1=1bull Many other contention scenarios possible

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313509831351

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1682

ROW CACHE LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

16

Elapsed and CPU times vs number of threadsOracle Database Appliance (24 SMT)

Latch wait

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1782

HOW ORACLEREQUESTS THE LATCH

RDTEX | Latch internals | RUOUG Seminar17

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1882

DTRACE REVEALS LATCH INTERFACE ROUTINES

Oracle calls the following functions to acquire the latch

ndash kslgetl(laddr wait why where) - get exclusive latch ndash kslg2c(l1l2trcwhy where) - get two excl child latches

ndash kslgetsl (laddrwaitwhywheremode) - get shared latch

bull 11g - ksl_get_shared_latch (hellip)

ndash kslg2cs(l1l2modetrcwhy where)) - get two shr child latches

ndash kslgpl(laddrcommentwhywhere) - get parent and all childs

ndash kslfre(laddr) - free the latch

Oracle give us possibility to do the same by oradebug call

RDTEX | Latch internals | RUOUG Seminar

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313501

983125983155983141 32983085983138983145983156 983152983151983154983156983155

64983138983145983156 O983154983137983139983148983141 983139983151983154983154983157983152983156983155 983156983144983141 983137983154983143983157983149983141983150983156983155 983151983142 983151983154983137983140983141983138983157983143 983139983137983148983148 983145983150 10204 983137983150983140 983137983138983151983158983141

18

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1982

ldquoWHERErdquo AND ldquoWHYrdquo ORACLE GETS THE LATCH

To request the latch Oracle kernel routines need

bull laddres ndash address of latch in SGAbull wait ndash flag for no-wait (0) or wait (1) latch acquisition

bull where ndash integer code for location from where the latch is acquired

Oracle lists possible latch ldquowhererdquo values in x$ksllwindxbull why - context of why the latch is acquiring at this ldquowhererdquo It

contains DBA for CBC latches (Tanel Poder) SGA chunk addressfor shared pool latch session address etchellip ldquoWhyrdquo meaning for

some ldquowhererdquo can be found in x$ksllwksllwlbl

bull mode ndash requesting state for shared lathes

ndash 8 SHARED mode 16 EXCLUSIVE mode

RDTEX | Latch internals | RUOUG Seminar

19

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2082

LIST OF LATCH ldquoWHERErdquo LOCATIONS

X$KSLLW ndash List of latch [W]here Part of v$latch_misses

ndash ksllwnam ndash Kernel function name(ldquokghupr1rdquo ldquoksqdelrdquo etc) ndash ksllwlbl ndash Description of ldquowhyrdquo context

Join this code on indx to

ndash x$ksuprlatksulawhr (v$latchholder)ndash ldquowhererdquo value that wasused by latch holder

ndash x$kslltkslltwhr (v$latch) ndash ldquowhererdquo that was used by lastsuccessful latch acquisition

ndash x$ksuprksllawer (v$process) ndash ldquowhererdquo that was used byprocess is being waiting for latch

ndash x$kslwscindx (v$latch_misses) ndash latch sleep statistics by

ldquowhererdquo

RDTEX | Latch internals | RUOUG Seminar

20

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2182

V$LATCH_MISSES

bull Based on X$KSLWSC - No[W]ait and [S]leep [C]ount stats byContext Documentation is rather obscure

bull The following counters are incremented on each latch sleep

sleep_count (ldquowhererdquo the latch was held)

wtr_slp_count (ldquowhererdquo the latch get was requested)

bull nwfail_count is incremented only for session which SID is equalto _LATCH_MISS_STAT_SID parameter

bull longhold_count is incremented for shared latches only when

someone held the latch at this ldquowhererdquo for the entire duration ofsleep

RDTEX | Latch internals | RUOUG Seminar

21

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2282

LATCH IS HOLDING BY PROCESS NOT SESSION

bull

RDTEX | Latch internals | RUOUG Seminar

983123983156983154983157983139983156 983147983155983157983152983154 983163

991270

983123983156983154983157983139983156 983147983155983148983148983137983163983147983155983148983148983156 98314798315598314898314898313798314898313798315698313114983133

983165

991270983165

983120983154983151983139983141983155983155 983142983145983160983141983140 983137983154983154983137983161

983158$983152983154983151983139983141983155983155 983085gt 983160$983147983155983157983152983154

983123983156983154983157983139983156 983147983155983148983148983156983163

991270

983165

L983145983155983156 983151983142 983137983148983148 983148983137983156983139983144983141983155

983158$983148983137983156983139983144 983085gt983160$983147983155983148983148983156

Each process has an array of references to the latches it is holding

22

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2382

KSLLA IS HIDDEN IN V$PROCESS

bull The process latching info is the kslla structure embedded in theprocess state object

bull Some kslla fields were externalized to SQL

bull kslla contains array of latches holding by process at each level andinformation about the state of any current attempt to acquire a

latch Not externalized directly

bull When you query v$latchholder x$ksuprlat Oracle scansv$processx$ksupr in order to find all the processes that are

holding the latchesbull Tanel Poder uses high frequency sampling of v$latchholder to

systematically troubleshoot latch contention

RDTEX | Latch internals | RUOUG Seminar

23

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2482

X$KSUPRKSLLA FIELDS INSTRUMENT THE LATCH GET

bull ksllalaq ndash address of latch acquiring Populated during immediateget (and spin before 11g)

bull ksllawat - latch being waited for This is v$processlatchwait

bull ksllawhy ndash ldquowhyrdquo for the latch being waited for

bull ksllawere ndash ldquowhererdquo for the latch being waited for

bull ksllalow ndash bit array of levels of currently holding latches

bull ksllaspn - latch this process is spinning on v$processlatchspinNot populated since 81

bull ksllaps - inter-process post statistics

RDTEX | Latch internals | RUOUG Seminar

24

Generic ldquolatch freerdquo and 27 latch specific ldquolatch free rdquo wait events areassociated with sleep only Spin is invisible to Oracle Wait Interface

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2582

THE LATCH STRUCTURE ndash KSLLT

RDTEX | Latch internals | RUOUG Seminar

struct ksllt

ltLatchgt

ldquowhererdquo and ldquowhyrdquo

Level latch class other attributes

Statistics

Latch wait list header

hellip

25

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2682

LATCH SIZE BY VERSION

nix 32bit nix 64bit Windows 32bit

734 92 - 120

806 104 - 104817 104 144 104

901 200 160

920 196 240 200

1010 256 208

1020 - 11203 100 160 104

RDTEX | Latch internals | RUOUG Seminar

x$ksmfsv ndash list of all fixed SGA variables983123E983116EC983124 DI983123983124I983118C983124 983147983155983149983142983155983155983145983162

F983122983119983117 983160$983147983155983149983142983155983158

983127HE983122E 983147983155983149983142983155983156983161983152 = 983147983155983148983148983156

Latch structure was bigger in 101 due to additional latch statistics

26

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2782

ORACLE LATCH IS NOT JUST A SINGLE MEMORY LOCATION

bull Before 11g Value of first latch byte (word for shared latches) wasused to determine latch state

0x00 ndash latch is free

0xFF ndash exclusive latch is busy Was 0x01 in Oracle 7

0x010x02hellip - shared latch holding by 12 hellip processes simultaneously

0x20000000 | pid - shared latch holding exclusively

bull In 11g first latch word shows the pid of the latch holder

0x00 ndash latch is free0x12 ndash Oracle process with pid 18 holds the exclusive latch

RDTEX | Latch internals | RUOUG Seminar

27

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2882

BUSY ORACLE 11G LATCH IN MEMORY

bull 112 Exclusive latch (32-bit platform)

bull 112 Shared latch EXCLUSIVE mode

RDTEX | Latch internals | RUOUG Seminar

SQLgt oradebug peek 200222A0 24

[200222A0 200222B8) = 00000016 00000001 000001D0 00000007

pid^ gets latch level

SQLgt oradebug peek 0xampladdr 24

[2000C814 2000C82C) = 2000001F 0000007C 0000007E 00000006 pid^ gets latch level

983139983151983150983156 983155983153983148983152983148983157983155 983150983151983148983151983143 9831409831419831499831519831350128

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2982

LATCH FIXED TABLES

bull Before Oracle 11g all the latches were externalized to SQL inv$latch (x$ksllt)

bull In 11g parent (fixed SGA) latches are externalized inv$latch_parent (x$kslltr_parent)

bull In 11g child (allocated on startup) latches are externalized in

v$latch_children (x$kslltr_children) bull Oracle 11g v$latch (x$kslltr) merges these data Beware select

from x$kslltr acquires all parent latches and their children

bull Only part of latch structure is visible in x$ksllthellip fixed tables ndash kslltwhr kslltwhy ndash ldquowhererdquo and ldquowhyrdquo latch was acquired last time

ndash kslltwgt hellip - statistics

ndash kslltlvl hellip - attributes

ndash kslltwtt ndash latch wait time

RDTEX | Latch internals | RUOUG Seminar

29

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3082

CUMULATIVE V$LATCH STATISTIC COUNTERS

RDTEX | Latch internals | RUOUG Seminar

Statistics x$ksllt When and how it increments

GETS kslltwgt ldquo++rdquo after wait mode latch get

MISSES kslltwff ldquo++rdquo after wait get if it was missed

SLEEPS kslltwsl ldquo+number_of_sleepsrdquo during get

SPIN_GETS ksllthst0 ldquo++rdquo if get was missed but not slept

WAIT_TIME kslltwtt ldquo+wait_timerdquo after latch get

IMMEDIATE_GETS kslltngt ldquo++rdquo after nowait mode latch get Is notincremented on miss

IMMEDIATE_MISSES kslltnfa ldquo++rdquo if nowait mode get was missed

30

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3182

MANY STATISTICS WERE LOST IN 102

In 102 Oracle made obsolete many useful latch statistics

Latch size was decreased and the following fields always contain 0now in v$latch

ndash v$latchwaiters_woken

ndash v$latchwaits_holding_latch

ndash v$latchsleep1hellipsleep10

ndash and many undocumented statistics

This was mentioned in 102 Reference

RDTEX | Latch internals | RUOUG Seminar

31

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3282

X$KSLLD - LATCH DESCRIPTORS

bull Flags and attributes describing each latch are stored in kslldt arrayof kslld structures

bull [K]ernel [S]ervice [L]atch [L]ock [D]escriptor

bull Some but not all of kslld fields are externalized via x$kslld andv$latchname

bull Located in the PGA

RDTEX | Latch internals | RUOUG Seminar

32

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382

LATCH ATTRIBUTES

bull Each latch have at least the following attributes in kslldt

ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared

ndash PAR Is the latch Solitary or Parent for the family of child latches

ndash G2C Can two child latches be simultaneously requested in wait mode

using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92

ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above

ndash Level 0-14 Used to prevent latch deadlocks

ndash Class 0-7 Spin and wait class assigned to the latch 92 and above

RDTEX | Latch internals | RUOUG Seminar

33

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482

LATCH ATTRIBUTES BY ORACLE VERSION

RDTEX | Latch internals | RUOUG Seminar

11203 553 154 72 - 6 93

Oracle Number of latches PAR G2C LNG UFS SHARED

7340 53 14 2 3 - -

8063 80 21 7 3 - 38174 152 48 19 4 - 9

9208 242 79 37 - - 19

10202 385 114 55 - 4 47

10203 388 117 58 - 4 48

10204 394 117 59 - 4 50

11106 496 145 67 - 6 81

11107 502 145 67 - 6 83

11201 535 149 70 - 6 86

11202 551 153 72 - 6 91

34

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582

LATCH LEVELS

bull Every latch has a level in the range 0-14 to avoid deadlocks

bull Process can request a latch X in wait mode after obtaining latch Y if and

only if level X gt level Y

bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y

bull If these rules are broken the Oracle process raises

ORA-600[504] - trying to obtain another latch at incompatible level Note 281041

ORA-600[525] - trying to obtain another latch at the same level Note 1388881

ORA-600[526] - latch child order violation Note 1388891

RDTEX | Latch internals | RUOUG Seminar

35

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682

PARENT AND CHILD LATCHES

bull Latch can be a parent child or solitary latch

bull Both parent and child latches share the same latch name

bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss

bull Child latches are allocated during instance startup

bull The parent latch can be gotten independently

bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s

RDTEX | Latch internals | RUOUG Seminar

(latch info) wait_event=0 bits=10

holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl

Context saved from call 0

state=busy

36

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782

LATCH TREES

ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches

Direct SGA access program output for 9206 instance with too smallshared pool

RDTEX | Latch internals | RUOUG Seminar

ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1

waiter ospid 129 sid 72 pid 45

holding a154b7120 library cache (157) level=5 child=17 kglupc child

waiter ospid 18255 sid 65 pid 930

waiter ospid 6690 sid 554 pid 1654

waiter ospid 4685 sid 879 pid 1034

hellip

waiter ospid 29749 sid 180 pid 155

holding a154b7db8 library cache (157) level=5 child=4 kglupc child

waiter ospid 13104 sid 281 pid 220

waiter ospid 24089 sid 565 pid 636

waiter ospid 25002 sid 621 pid 1481

waiter ospid 16930 sid 1046 pid 783hellip

37

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882

LATCH CLASSES

bull Classes with different spin and wait policies

bull Up to 8 classes By default all the latches belong to class 0

bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2

bull Latch can be assigned to class by its number For example

ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000

ndash _LATCH_CLASSES=46

bull Will be discussed later

RDTEX | Latch internals | RUOUG Seminar

SQLgt select from x$ksllclass

INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4

---- ------ ----- -------- ------ ------ ------ ------ ------

0 20000 0 1 8000 8000 8000 8000 8000

hellip

6 100 12 1000 2000 3000 4000 4000 4000

38

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982

LATCH ACQUISITION IN NO-WAIT MODE

bull kslgetl(laddr0whywhere) ndash Exclusive latch get

bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep

ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches

ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches

bull IMMEDIATE_ statistics are updated

bull Used when many candidate latches are available to bypass the

latch level rule or as part of CBC chain examination

RDTEX | Latch internals | RUOUG Seminar

sqlplus nolog demo_03sql39

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082

LATCH WAITS

RDTEX | Latch internals | RUOUG Seminar 40

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182

WAITING FOR THE LATCH

RDTEX | Latch internals | RUOUG Seminar

983123 G A

C983120983125 1 C983120983125 2

983120983154983151983139983141983155983155 A

983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)

983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144

983116983137983156983139983144

983120983154983151983139983141983155983155 B

41

RDTEX | L h i l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282

LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION

bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries

again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of

times

bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep

bull A sleep corresponds to a wait for the latch free wait event

bull Initially a process sleeps for 001 s This time is doubled in every

subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip

RDTEX | Latch internals | RUOUG Seminar

Still did not found any versionwhich used this

May be it was lt= Oracle 6

42

RDTEX | L t h i t l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382

LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81

Latch gets with wait (kslgetl(laddress1hellip))

ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times

ndash Sleep on latch free event with exponential backoff

ndash Repeat

RDTEX | Latch internals | RUOUG Seminar

43

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482

8I LATCH SPIN CODE FLOW USING DTRACE

RDTEX | Latch internals | RUOUG Seminar

bull hellip Event 10046 trace

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2

983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29

983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270

sqlplus nolog demo_04_8sql 29

44

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582

POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF

bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec

RDTEX | Latch internals | RUOUG Seminar

12 ]2 )1[(

minus= +wait N

timeout

45

bull Most waits were for nothing ndash latch already was free

bull Latch utilization can not be high

bull Unnecessary spins provoke CPU thrashing

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682

92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE

sqlplus nolog demo_05sql 45

RDTEX | Latch internals | RUOUG Seminar

kslgetl(0x50006318 1)

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt kslges(0x50006318 ) -wait latch get

-gt skgslsgts(0x50006318 ) -spin latch get

-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT

-gt kskthbwt(0x0)

-gt kslwlmod() - set up Wait List

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt skgpwwait -sleep latch getsemop(11 17-10 1)

983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140

46

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782

RELIABLE LATCH WAITS

bull Hidden latch wait revolution that we missed Latch relies on waitposting

bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout

bull Exclusive latch spins 20000 cycles by default

bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock

bull Latches assigned to non-default class wait until timeout

bull Version 90 was transient and used mix of backoff and wait posting

This is why _LATCH_WAIT_POSTING parameter disappeared in9i

bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin

RDTEX | Latch internals | RUOUG Seminar

47

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882

IMAGINE THE TIME MICROSCOPE

| a a s | a

Reality One million zoomed time

1 us 1 sec

1 ms 17 min1 sec 115 days

Light speed (300000 kmsec) Sonic speed (300 msec)

CPU tick (2 GHz) 00005 sec

Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec

Max spin time for shared latch (~220005 ticks) ~10 sec

Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec

Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec

OS context switch time (10us-1ms) 10sec-17min

48

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982

LATCH WAITS UNDER TIME MICROSCOPE

Old 8i latch algorithmSpin for latch during 5 seconds

Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min

hellipSpin again for 5 seconds

Sleep again for 23 days (2 sec) hellip

New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)

If spin not succeeded go to sleep until post

According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)

| |

49

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082

_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES

bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and

_LATCH_CLASS_0 parameters

bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000

bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter

bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch

50

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182

TIMED LATCH WAIT POSTING

bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel

bull This can lead to instance hang ndash process will never be woken up

bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout

bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem

bull Generic AIX ports always use timed latch wait posting

bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux

51

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282

FINE GRAIN LATCH TUNING

bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting

bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7

bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo

microseconds using pollsys() (not semtimedop()) system call

ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times

Loop up to ldquoSpinsrdquo cycles

Yield CPU using yield() (or sched_yield())

ndash Sleep for ldquoSleepXrdquo usecs

ndash Then spin again hellip

52

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382

TRACES OF NONDEFAULT CLASS LATCH

983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152

991270

9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000

991270 E983158983141983150983156 10046 983156983154983137983139983141

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482

PERFORMANCE OF LATCH CLASSES

Row cache objects latch contention testcase on ODA (24 SMT)54

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582

IS THE _SPIN_COUNT REALLY STATIC

bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch

waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used

ldquocache buffers chainrdquo latch contention

bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass

(_LATCH_CLASS_X) By default ndash 20000

bull Shared latch spin can be tuned by _SPIN_COUNT value By

default ndash 2000 However this is not trivial

55

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682

SHARED LATCH BEHAVE LIKE ENQUEUE

bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80

bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests

bull This blocking achieved by special 0x40000000 bit in the latch

value This bit is a flag that some incompatible latch acquisitionis in progress

bull Queued processes will be woken up one by one on latch release

bull X mode latch gets effectively serialize the shared latch

bull See my blog for description of experiments

56

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782

TRACE OF SHARED LATCH WAIT

Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the

latch wait list and then spins again

983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141

983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156

983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983147983155983148983159983148983149983151983140()

983147983155983148983155983147983143983155(098316050009BAC )

983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140

983155983141983149983151983152(27)

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882

SHARED LATCH ACQUISITION

bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned

bull X mode shared latch get spins by default up to 4000 cycles

bull S mode does not spin at all (or spins in unknown way)

S mode get X mode get

Held in S mode Compatible 2_SPIN_COUNT

Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT

58

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982

LATCH RELEASE

bull Free the latch ndash kslfre(laddr)

bull Oracle process releases the latch nonatomically

bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process

bull This requires less bus invalidation and ensures propagation of latch

release to other local cachesbull Not fair policy Spinners on the local CPU board have the

preference

bull Then process posts the first process in the list of waiters

59

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082

LATCH CONTENTION

RDTEX | Latch internals | RUOUG Seminar60

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182

LATCH CONTENTION

bull Occurs when a latch is required by several processes at thesame time

bull Mostly due to bugs or abnormal application behavior

bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp

bull To diagnose the latch contention we need to investigate thelatch statistics [8]

61

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282

DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS

Latch requests arrival rate

Immediate gets efficiency

Latch sleeps ratio

Latch wait time per second

Latch spin efficiencymisses

getsspin∆

∆= _σ

misses

sleepsK

∆=

time

gets

∆=λ

gets

misses

∆= ρ

time

timewait W

∆=

_

Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics

62

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382

SAMPLED AVERAGES

By sampling of x$ksupr we can estimate

ndash Length of latch wait queue

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS

ndash Number of spinning processes in 10g

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS

Sampling of v$lathholder estimates latch utilization

Using DTrace we can measure

ndash Latch acquisition and holding times

ndash Actual spin count

w L

s N

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482

DERIVED LATCH STATISTICS

Latch utilization (PASTA)

Average holding time

Length of latch wait list

Recurrent sleeps ratio

Latch acquisition time

time

timeholdinglatchU

∆=asymp

__ ρ

Re_100

___

questsGet

TimeSnap MissGet Pct S

lowast==

λ

W L =

κ

κ σ 1minus+

)(1

W N T saq += minusλ

64

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 8: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 882

HOW SPINLOCK WORKS

RDTEX | Latch internals | RUOUG Seminar

8

CPU3

Free FreeHolding by CPU 1

Spin polls the spinlock location

Get

Miss Sleep Spin count ( )∆

Holding time (S )

CPU2 Spin Holding the spinlock

Holding by CPU 2

CPU1 Holding the spinlock

The spinlock location

Sleep releases the CPU

bull Oracle latches

ndash Use atomic hardware instruction for Immediate get

ndash If missed process spins by polling latch location during Spin get

ndash Number of spin cycles is bounded by spin count

ndash In spin get not succeed the process acquiring the latch sleeps

hellip

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 982

CLASSIC SPINLOCKS

bull Wiki hellip spinlock hellip waits in a loop repeatedly checking until the lockbecomes available hellip

bull Introduced by Edsger Dijkstra [4] in 1965 Have been thoroughlyinvestigated since that time [5]

bull Many sophisticated spinlock realizations were proposed and evaluated

(TS TTS Delay MCS Anderson)

bull Two general types

ndash System spinlocks Kernel OS threads cannot wait Major metrics

atomic operations frequency Shared bus utilization

ndash User spinlocks Oracle latch and mutex Average lock holding time ~10 us It is more efficient to poll a lock rather than pre-empt the threaddoing 1 ms context switch Metrics CPU and elapsed times

RDTEX | Latch internals | RUOUG Seminar

9

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1082

THE NEED FOR SPIN

RDTEX | Latch internals | RUOUG Seminar

10

Time to complete CBC latch contention testcase vs number of threads

No spinning _spin_count=0

_spin_count=2000

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1182

SPINLOCK REALIZATIONS

RDTEX | Latch internals | RUOUG Seminar

Spinlock Pseudocode Problems

TS

pre-112 mutex

while(Test_and_Set(lock)) Bus saturation

TTS

Oracle latch

while(lock||Test_and_Set(lock)) Invalidation storms onrelease (ldquoopen doorrdquo

ldquothundering herdsrdquo)

Delay

Latch and 112

Mutex

Adjustable delay Slower under contention

Anderson MCSetc

Queues Widely used in JavaLinux kernel hellip

CPU and memoryoverhead preemptionissues

11

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1282

THE TOOLS

RDTEX | Latch internals | RUOUG Seminar12

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1382

DTRACE SOLARIS DYNAMIC TRACING FRAMEWORK

DTrace allows us to investigate how Oracle latches perform in real time ndash Create triggers on any event inside Solaris and function call inside Oracle

providermodulefunctionname

pid1910oraclekslgetlentry ndash Write trigger bodies ndash actions May read and change any address location ndash Can count the latch spins trace the latch waits perform experiments ndash Measure times and distributions up to microseconds precision

RDTEX | Latch internals | RUOUG Seminar

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313500

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1482

ORADEBUG INTERNAL ORACLE DEBUGGER

RDTEX | Latch internals | RUOUG Seminar

bull oradebug call Allows us invoke any internal Oracle function manually

SQLgt oradebug peek 200222A0 24

[200222A0 200222B8) = 00000016 00000001 000001D0 00000007

SQLgt oradebug call kslgetl 0x5001A020 1 2 3Function returned 1

bull oradebug peek Examines contents of any memory location

bull oradebug poke Modifies memory

bull oradebug watch Watch a region of memory

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1582

LATCH CONTENTION TESTCASES

RDTEX | Latch internals | RUOUG Seminar

15

create table latch_contention as select from dualhellipfor i in 11000000loopselect dummy into a from latch_contention

end loop

bull Exclusive and shared latches behave differentlybull Contention for shared cache buffer chains latch arises when manysessions scan the same block concurrently and _fastpin_enable=0

bull To induce the contention for exclusive ldquorow cache objectsrdquo latch protectingdc_users row cache just add several dynamic RLS policies

exec dbms_rlsadd_policy(systemlatch_contentionrls_1 hellip

exec dbms_rlsadd_policy(systemlatch_contentionrls_2 helliphellip

bull These dummy policies generate predicates 1=1bull Many other contention scenarios possible

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313509831351

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1682

ROW CACHE LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

16

Elapsed and CPU times vs number of threadsOracle Database Appliance (24 SMT)

Latch wait

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1782

HOW ORACLEREQUESTS THE LATCH

RDTEX | Latch internals | RUOUG Seminar17

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1882

DTRACE REVEALS LATCH INTERFACE ROUTINES

Oracle calls the following functions to acquire the latch

ndash kslgetl(laddr wait why where) - get exclusive latch ndash kslg2c(l1l2trcwhy where) - get two excl child latches

ndash kslgetsl (laddrwaitwhywheremode) - get shared latch

bull 11g - ksl_get_shared_latch (hellip)

ndash kslg2cs(l1l2modetrcwhy where)) - get two shr child latches

ndash kslgpl(laddrcommentwhywhere) - get parent and all childs

ndash kslfre(laddr) - free the latch

Oracle give us possibility to do the same by oradebug call

RDTEX | Latch internals | RUOUG Seminar

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313501

983125983155983141 32983085983138983145983156 983152983151983154983156983155

64983138983145983156 O983154983137983139983148983141 983139983151983154983154983157983152983156983155 983156983144983141 983137983154983143983157983149983141983150983156983155 983151983142 983151983154983137983140983141983138983157983143 983139983137983148983148 983145983150 10204 983137983150983140 983137983138983151983158983141

18

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1982

ldquoWHERErdquo AND ldquoWHYrdquo ORACLE GETS THE LATCH

To request the latch Oracle kernel routines need

bull laddres ndash address of latch in SGAbull wait ndash flag for no-wait (0) or wait (1) latch acquisition

bull where ndash integer code for location from where the latch is acquired

Oracle lists possible latch ldquowhererdquo values in x$ksllwindxbull why - context of why the latch is acquiring at this ldquowhererdquo It

contains DBA for CBC latches (Tanel Poder) SGA chunk addressfor shared pool latch session address etchellip ldquoWhyrdquo meaning for

some ldquowhererdquo can be found in x$ksllwksllwlbl

bull mode ndash requesting state for shared lathes

ndash 8 SHARED mode 16 EXCLUSIVE mode

RDTEX | Latch internals | RUOUG Seminar

19

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2082

LIST OF LATCH ldquoWHERErdquo LOCATIONS

X$KSLLW ndash List of latch [W]here Part of v$latch_misses

ndash ksllwnam ndash Kernel function name(ldquokghupr1rdquo ldquoksqdelrdquo etc) ndash ksllwlbl ndash Description of ldquowhyrdquo context

Join this code on indx to

ndash x$ksuprlatksulawhr (v$latchholder)ndash ldquowhererdquo value that wasused by latch holder

ndash x$kslltkslltwhr (v$latch) ndash ldquowhererdquo that was used by lastsuccessful latch acquisition

ndash x$ksuprksllawer (v$process) ndash ldquowhererdquo that was used byprocess is being waiting for latch

ndash x$kslwscindx (v$latch_misses) ndash latch sleep statistics by

ldquowhererdquo

RDTEX | Latch internals | RUOUG Seminar

20

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2182

V$LATCH_MISSES

bull Based on X$KSLWSC - No[W]ait and [S]leep [C]ount stats byContext Documentation is rather obscure

bull The following counters are incremented on each latch sleep

sleep_count (ldquowhererdquo the latch was held)

wtr_slp_count (ldquowhererdquo the latch get was requested)

bull nwfail_count is incremented only for session which SID is equalto _LATCH_MISS_STAT_SID parameter

bull longhold_count is incremented for shared latches only when

someone held the latch at this ldquowhererdquo for the entire duration ofsleep

RDTEX | Latch internals | RUOUG Seminar

21

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2282

LATCH IS HOLDING BY PROCESS NOT SESSION

bull

RDTEX | Latch internals | RUOUG Seminar

983123983156983154983157983139983156 983147983155983157983152983154 983163

991270

983123983156983154983157983139983156 983147983155983148983148983137983163983147983155983148983148983156 98314798315598314898314898313798314898313798315698313114983133

983165

991270983165

983120983154983151983139983141983155983155 983142983145983160983141983140 983137983154983154983137983161

983158$983152983154983151983139983141983155983155 983085gt 983160$983147983155983157983152983154

983123983156983154983157983139983156 983147983155983148983148983156983163

991270

983165

L983145983155983156 983151983142 983137983148983148 983148983137983156983139983144983141983155

983158$983148983137983156983139983144 983085gt983160$983147983155983148983148983156

Each process has an array of references to the latches it is holding

22

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2382

KSLLA IS HIDDEN IN V$PROCESS

bull The process latching info is the kslla structure embedded in theprocess state object

bull Some kslla fields were externalized to SQL

bull kslla contains array of latches holding by process at each level andinformation about the state of any current attempt to acquire a

latch Not externalized directly

bull When you query v$latchholder x$ksuprlat Oracle scansv$processx$ksupr in order to find all the processes that are

holding the latchesbull Tanel Poder uses high frequency sampling of v$latchholder to

systematically troubleshoot latch contention

RDTEX | Latch internals | RUOUG Seminar

23

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2482

X$KSUPRKSLLA FIELDS INSTRUMENT THE LATCH GET

bull ksllalaq ndash address of latch acquiring Populated during immediateget (and spin before 11g)

bull ksllawat - latch being waited for This is v$processlatchwait

bull ksllawhy ndash ldquowhyrdquo for the latch being waited for

bull ksllawere ndash ldquowhererdquo for the latch being waited for

bull ksllalow ndash bit array of levels of currently holding latches

bull ksllaspn - latch this process is spinning on v$processlatchspinNot populated since 81

bull ksllaps - inter-process post statistics

RDTEX | Latch internals | RUOUG Seminar

24

Generic ldquolatch freerdquo and 27 latch specific ldquolatch free rdquo wait events areassociated with sleep only Spin is invisible to Oracle Wait Interface

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2582

THE LATCH STRUCTURE ndash KSLLT

RDTEX | Latch internals | RUOUG Seminar

struct ksllt

ltLatchgt

ldquowhererdquo and ldquowhyrdquo

Level latch class other attributes

Statistics

Latch wait list header

hellip

25

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2682

LATCH SIZE BY VERSION

nix 32bit nix 64bit Windows 32bit

734 92 - 120

806 104 - 104817 104 144 104

901 200 160

920 196 240 200

1010 256 208

1020 - 11203 100 160 104

RDTEX | Latch internals | RUOUG Seminar

x$ksmfsv ndash list of all fixed SGA variables983123E983116EC983124 DI983123983124I983118C983124 983147983155983149983142983155983155983145983162

F983122983119983117 983160$983147983155983149983142983155983158

983127HE983122E 983147983155983149983142983155983156983161983152 = 983147983155983148983148983156

Latch structure was bigger in 101 due to additional latch statistics

26

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2782

ORACLE LATCH IS NOT JUST A SINGLE MEMORY LOCATION

bull Before 11g Value of first latch byte (word for shared latches) wasused to determine latch state

0x00 ndash latch is free

0xFF ndash exclusive latch is busy Was 0x01 in Oracle 7

0x010x02hellip - shared latch holding by 12 hellip processes simultaneously

0x20000000 | pid - shared latch holding exclusively

bull In 11g first latch word shows the pid of the latch holder

0x00 ndash latch is free0x12 ndash Oracle process with pid 18 holds the exclusive latch

RDTEX | Latch internals | RUOUG Seminar

27

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2882

BUSY ORACLE 11G LATCH IN MEMORY

bull 112 Exclusive latch (32-bit platform)

bull 112 Shared latch EXCLUSIVE mode

RDTEX | Latch internals | RUOUG Seminar

SQLgt oradebug peek 200222A0 24

[200222A0 200222B8) = 00000016 00000001 000001D0 00000007

pid^ gets latch level

SQLgt oradebug peek 0xampladdr 24

[2000C814 2000C82C) = 2000001F 0000007C 0000007E 00000006 pid^ gets latch level

983139983151983150983156 983155983153983148983152983148983157983155 983150983151983148983151983143 9831409831419831499831519831350128

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2982

LATCH FIXED TABLES

bull Before Oracle 11g all the latches were externalized to SQL inv$latch (x$ksllt)

bull In 11g parent (fixed SGA) latches are externalized inv$latch_parent (x$kslltr_parent)

bull In 11g child (allocated on startup) latches are externalized in

v$latch_children (x$kslltr_children) bull Oracle 11g v$latch (x$kslltr) merges these data Beware select

from x$kslltr acquires all parent latches and their children

bull Only part of latch structure is visible in x$ksllthellip fixed tables ndash kslltwhr kslltwhy ndash ldquowhererdquo and ldquowhyrdquo latch was acquired last time

ndash kslltwgt hellip - statistics

ndash kslltlvl hellip - attributes

ndash kslltwtt ndash latch wait time

RDTEX | Latch internals | RUOUG Seminar

29

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3082

CUMULATIVE V$LATCH STATISTIC COUNTERS

RDTEX | Latch internals | RUOUG Seminar

Statistics x$ksllt When and how it increments

GETS kslltwgt ldquo++rdquo after wait mode latch get

MISSES kslltwff ldquo++rdquo after wait get if it was missed

SLEEPS kslltwsl ldquo+number_of_sleepsrdquo during get

SPIN_GETS ksllthst0 ldquo++rdquo if get was missed but not slept

WAIT_TIME kslltwtt ldquo+wait_timerdquo after latch get

IMMEDIATE_GETS kslltngt ldquo++rdquo after nowait mode latch get Is notincremented on miss

IMMEDIATE_MISSES kslltnfa ldquo++rdquo if nowait mode get was missed

30

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3182

MANY STATISTICS WERE LOST IN 102

In 102 Oracle made obsolete many useful latch statistics

Latch size was decreased and the following fields always contain 0now in v$latch

ndash v$latchwaiters_woken

ndash v$latchwaits_holding_latch

ndash v$latchsleep1hellipsleep10

ndash and many undocumented statistics

This was mentioned in 102 Reference

RDTEX | Latch internals | RUOUG Seminar

31

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3282

X$KSLLD - LATCH DESCRIPTORS

bull Flags and attributes describing each latch are stored in kslldt arrayof kslld structures

bull [K]ernel [S]ervice [L]atch [L]ock [D]escriptor

bull Some but not all of kslld fields are externalized via x$kslld andv$latchname

bull Located in the PGA

RDTEX | Latch internals | RUOUG Seminar

32

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382

LATCH ATTRIBUTES

bull Each latch have at least the following attributes in kslldt

ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared

ndash PAR Is the latch Solitary or Parent for the family of child latches

ndash G2C Can two child latches be simultaneously requested in wait mode

using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92

ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above

ndash Level 0-14 Used to prevent latch deadlocks

ndash Class 0-7 Spin and wait class assigned to the latch 92 and above

RDTEX | Latch internals | RUOUG Seminar

33

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482

LATCH ATTRIBUTES BY ORACLE VERSION

RDTEX | Latch internals | RUOUG Seminar

11203 553 154 72 - 6 93

Oracle Number of latches PAR G2C LNG UFS SHARED

7340 53 14 2 3 - -

8063 80 21 7 3 - 38174 152 48 19 4 - 9

9208 242 79 37 - - 19

10202 385 114 55 - 4 47

10203 388 117 58 - 4 48

10204 394 117 59 - 4 50

11106 496 145 67 - 6 81

11107 502 145 67 - 6 83

11201 535 149 70 - 6 86

11202 551 153 72 - 6 91

34

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582

LATCH LEVELS

bull Every latch has a level in the range 0-14 to avoid deadlocks

bull Process can request a latch X in wait mode after obtaining latch Y if and

only if level X gt level Y

bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y

bull If these rules are broken the Oracle process raises

ORA-600[504] - trying to obtain another latch at incompatible level Note 281041

ORA-600[525] - trying to obtain another latch at the same level Note 1388881

ORA-600[526] - latch child order violation Note 1388891

RDTEX | Latch internals | RUOUG Seminar

35

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682

PARENT AND CHILD LATCHES

bull Latch can be a parent child or solitary latch

bull Both parent and child latches share the same latch name

bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss

bull Child latches are allocated during instance startup

bull The parent latch can be gotten independently

bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s

RDTEX | Latch internals | RUOUG Seminar

(latch info) wait_event=0 bits=10

holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl

Context saved from call 0

state=busy

36

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782

LATCH TREES

ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches

Direct SGA access program output for 9206 instance with too smallshared pool

RDTEX | Latch internals | RUOUG Seminar

ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1

waiter ospid 129 sid 72 pid 45

holding a154b7120 library cache (157) level=5 child=17 kglupc child

waiter ospid 18255 sid 65 pid 930

waiter ospid 6690 sid 554 pid 1654

waiter ospid 4685 sid 879 pid 1034

hellip

waiter ospid 29749 sid 180 pid 155

holding a154b7db8 library cache (157) level=5 child=4 kglupc child

waiter ospid 13104 sid 281 pid 220

waiter ospid 24089 sid 565 pid 636

waiter ospid 25002 sid 621 pid 1481

waiter ospid 16930 sid 1046 pid 783hellip

37

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882

LATCH CLASSES

bull Classes with different spin and wait policies

bull Up to 8 classes By default all the latches belong to class 0

bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2

bull Latch can be assigned to class by its number For example

ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000

ndash _LATCH_CLASSES=46

bull Will be discussed later

RDTEX | Latch internals | RUOUG Seminar

SQLgt select from x$ksllclass

INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4

---- ------ ----- -------- ------ ------ ------ ------ ------

0 20000 0 1 8000 8000 8000 8000 8000

hellip

6 100 12 1000 2000 3000 4000 4000 4000

38

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982

LATCH ACQUISITION IN NO-WAIT MODE

bull kslgetl(laddr0whywhere) ndash Exclusive latch get

bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep

ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches

ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches

bull IMMEDIATE_ statistics are updated

bull Used when many candidate latches are available to bypass the

latch level rule or as part of CBC chain examination

RDTEX | Latch internals | RUOUG Seminar

sqlplus nolog demo_03sql39

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082

LATCH WAITS

RDTEX | Latch internals | RUOUG Seminar 40

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182

WAITING FOR THE LATCH

RDTEX | Latch internals | RUOUG Seminar

983123 G A

C983120983125 1 C983120983125 2

983120983154983151983139983141983155983155 A

983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)

983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144

983116983137983156983139983144

983120983154983151983139983141983155983155 B

41

RDTEX | L h i l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282

LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION

bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries

again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of

times

bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep

bull A sleep corresponds to a wait for the latch free wait event

bull Initially a process sleeps for 001 s This time is doubled in every

subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip

RDTEX | Latch internals | RUOUG Seminar

Still did not found any versionwhich used this

May be it was lt= Oracle 6

42

RDTEX | L t h i t l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382

LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81

Latch gets with wait (kslgetl(laddress1hellip))

ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times

ndash Sleep on latch free event with exponential backoff

ndash Repeat

RDTEX | Latch internals | RUOUG Seminar

43

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482

8I LATCH SPIN CODE FLOW USING DTRACE

RDTEX | Latch internals | RUOUG Seminar

bull hellip Event 10046 trace

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2

983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29

983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270

sqlplus nolog demo_04_8sql 29

44

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582

POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF

bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec

RDTEX | Latch internals | RUOUG Seminar

12 ]2 )1[(

minus= +wait N

timeout

45

bull Most waits were for nothing ndash latch already was free

bull Latch utilization can not be high

bull Unnecessary spins provoke CPU thrashing

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682

92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE

sqlplus nolog demo_05sql 45

RDTEX | Latch internals | RUOUG Seminar

kslgetl(0x50006318 1)

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt kslges(0x50006318 ) -wait latch get

-gt skgslsgts(0x50006318 ) -spin latch get

-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT

-gt kskthbwt(0x0)

-gt kslwlmod() - set up Wait List

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt skgpwwait -sleep latch getsemop(11 17-10 1)

983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140

46

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782

RELIABLE LATCH WAITS

bull Hidden latch wait revolution that we missed Latch relies on waitposting

bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout

bull Exclusive latch spins 20000 cycles by default

bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock

bull Latches assigned to non-default class wait until timeout

bull Version 90 was transient and used mix of backoff and wait posting

This is why _LATCH_WAIT_POSTING parameter disappeared in9i

bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin

RDTEX | Latch internals | RUOUG Seminar

47

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882

IMAGINE THE TIME MICROSCOPE

| a a s | a

Reality One million zoomed time

1 us 1 sec

1 ms 17 min1 sec 115 days

Light speed (300000 kmsec) Sonic speed (300 msec)

CPU tick (2 GHz) 00005 sec

Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec

Max spin time for shared latch (~220005 ticks) ~10 sec

Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec

Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec

OS context switch time (10us-1ms) 10sec-17min

48

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982

LATCH WAITS UNDER TIME MICROSCOPE

Old 8i latch algorithmSpin for latch during 5 seconds

Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min

hellipSpin again for 5 seconds

Sleep again for 23 days (2 sec) hellip

New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)

If spin not succeeded go to sleep until post

According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)

| |

49

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082

_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES

bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and

_LATCH_CLASS_0 parameters

bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000

bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter

bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch

50

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182

TIMED LATCH WAIT POSTING

bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel

bull This can lead to instance hang ndash process will never be woken up

bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout

bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem

bull Generic AIX ports always use timed latch wait posting

bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux

51

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282

FINE GRAIN LATCH TUNING

bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting

bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7

bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo

microseconds using pollsys() (not semtimedop()) system call

ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times

Loop up to ldquoSpinsrdquo cycles

Yield CPU using yield() (or sched_yield())

ndash Sleep for ldquoSleepXrdquo usecs

ndash Then spin again hellip

52

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382

TRACES OF NONDEFAULT CLASS LATCH

983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152

991270

9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000

991270 E983158983141983150983156 10046 983156983154983137983139983141

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482

PERFORMANCE OF LATCH CLASSES

Row cache objects latch contention testcase on ODA (24 SMT)54

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582

IS THE _SPIN_COUNT REALLY STATIC

bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch

waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used

ldquocache buffers chainrdquo latch contention

bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass

(_LATCH_CLASS_X) By default ndash 20000

bull Shared latch spin can be tuned by _SPIN_COUNT value By

default ndash 2000 However this is not trivial

55

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682

SHARED LATCH BEHAVE LIKE ENQUEUE

bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80

bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests

bull This blocking achieved by special 0x40000000 bit in the latch

value This bit is a flag that some incompatible latch acquisitionis in progress

bull Queued processes will be woken up one by one on latch release

bull X mode latch gets effectively serialize the shared latch

bull See my blog for description of experiments

56

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782

TRACE OF SHARED LATCH WAIT

Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the

latch wait list and then spins again

983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141

983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156

983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983147983155983148983159983148983149983151983140()

983147983155983148983155983147983143983155(098316050009BAC )

983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140

983155983141983149983151983152(27)

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882

SHARED LATCH ACQUISITION

bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned

bull X mode shared latch get spins by default up to 4000 cycles

bull S mode does not spin at all (or spins in unknown way)

S mode get X mode get

Held in S mode Compatible 2_SPIN_COUNT

Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT

58

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982

LATCH RELEASE

bull Free the latch ndash kslfre(laddr)

bull Oracle process releases the latch nonatomically

bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process

bull This requires less bus invalidation and ensures propagation of latch

release to other local cachesbull Not fair policy Spinners on the local CPU board have the

preference

bull Then process posts the first process in the list of waiters

59

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082

LATCH CONTENTION

RDTEX | Latch internals | RUOUG Seminar60

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182

LATCH CONTENTION

bull Occurs when a latch is required by several processes at thesame time

bull Mostly due to bugs or abnormal application behavior

bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp

bull To diagnose the latch contention we need to investigate thelatch statistics [8]

61

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282

DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS

Latch requests arrival rate

Immediate gets efficiency

Latch sleeps ratio

Latch wait time per second

Latch spin efficiencymisses

getsspin∆

∆= _σ

misses

sleepsK

∆=

time

gets

∆=λ

gets

misses

∆= ρ

time

timewait W

∆=

_

Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics

62

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382

SAMPLED AVERAGES

By sampling of x$ksupr we can estimate

ndash Length of latch wait queue

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS

ndash Number of spinning processes in 10g

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS

Sampling of v$lathholder estimates latch utilization

Using DTrace we can measure

ndash Latch acquisition and holding times

ndash Actual spin count

w L

s N

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482

DERIVED LATCH STATISTICS

Latch utilization (PASTA)

Average holding time

Length of latch wait list

Recurrent sleeps ratio

Latch acquisition time

time

timeholdinglatchU

∆=asymp

__ ρ

Re_100

___

questsGet

TimeSnap MissGet Pct S

lowast==

λ

W L =

κ

κ σ 1minus+

)(1

W N T saq += minusλ

64

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 9: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 982

CLASSIC SPINLOCKS

bull Wiki hellip spinlock hellip waits in a loop repeatedly checking until the lockbecomes available hellip

bull Introduced by Edsger Dijkstra [4] in 1965 Have been thoroughlyinvestigated since that time [5]

bull Many sophisticated spinlock realizations were proposed and evaluated

(TS TTS Delay MCS Anderson)

bull Two general types

ndash System spinlocks Kernel OS threads cannot wait Major metrics

atomic operations frequency Shared bus utilization

ndash User spinlocks Oracle latch and mutex Average lock holding time ~10 us It is more efficient to poll a lock rather than pre-empt the threaddoing 1 ms context switch Metrics CPU and elapsed times

RDTEX | Latch internals | RUOUG Seminar

9

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1082

THE NEED FOR SPIN

RDTEX | Latch internals | RUOUG Seminar

10

Time to complete CBC latch contention testcase vs number of threads

No spinning _spin_count=0

_spin_count=2000

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1182

SPINLOCK REALIZATIONS

RDTEX | Latch internals | RUOUG Seminar

Spinlock Pseudocode Problems

TS

pre-112 mutex

while(Test_and_Set(lock)) Bus saturation

TTS

Oracle latch

while(lock||Test_and_Set(lock)) Invalidation storms onrelease (ldquoopen doorrdquo

ldquothundering herdsrdquo)

Delay

Latch and 112

Mutex

Adjustable delay Slower under contention

Anderson MCSetc

Queues Widely used in JavaLinux kernel hellip

CPU and memoryoverhead preemptionissues

11

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1282

THE TOOLS

RDTEX | Latch internals | RUOUG Seminar12

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1382

DTRACE SOLARIS DYNAMIC TRACING FRAMEWORK

DTrace allows us to investigate how Oracle latches perform in real time ndash Create triggers on any event inside Solaris and function call inside Oracle

providermodulefunctionname

pid1910oraclekslgetlentry ndash Write trigger bodies ndash actions May read and change any address location ndash Can count the latch spins trace the latch waits perform experiments ndash Measure times and distributions up to microseconds precision

RDTEX | Latch internals | RUOUG Seminar

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313500

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1482

ORADEBUG INTERNAL ORACLE DEBUGGER

RDTEX | Latch internals | RUOUG Seminar

bull oradebug call Allows us invoke any internal Oracle function manually

SQLgt oradebug peek 200222A0 24

[200222A0 200222B8) = 00000016 00000001 000001D0 00000007

SQLgt oradebug call kslgetl 0x5001A020 1 2 3Function returned 1

bull oradebug peek Examines contents of any memory location

bull oradebug poke Modifies memory

bull oradebug watch Watch a region of memory

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1582

LATCH CONTENTION TESTCASES

RDTEX | Latch internals | RUOUG Seminar

15

create table latch_contention as select from dualhellipfor i in 11000000loopselect dummy into a from latch_contention

end loop

bull Exclusive and shared latches behave differentlybull Contention for shared cache buffer chains latch arises when manysessions scan the same block concurrently and _fastpin_enable=0

bull To induce the contention for exclusive ldquorow cache objectsrdquo latch protectingdc_users row cache just add several dynamic RLS policies

exec dbms_rlsadd_policy(systemlatch_contentionrls_1 hellip

exec dbms_rlsadd_policy(systemlatch_contentionrls_2 helliphellip

bull These dummy policies generate predicates 1=1bull Many other contention scenarios possible

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313509831351

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1682

ROW CACHE LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

16

Elapsed and CPU times vs number of threadsOracle Database Appliance (24 SMT)

Latch wait

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1782

HOW ORACLEREQUESTS THE LATCH

RDTEX | Latch internals | RUOUG Seminar17

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1882

DTRACE REVEALS LATCH INTERFACE ROUTINES

Oracle calls the following functions to acquire the latch

ndash kslgetl(laddr wait why where) - get exclusive latch ndash kslg2c(l1l2trcwhy where) - get two excl child latches

ndash kslgetsl (laddrwaitwhywheremode) - get shared latch

bull 11g - ksl_get_shared_latch (hellip)

ndash kslg2cs(l1l2modetrcwhy where)) - get two shr child latches

ndash kslgpl(laddrcommentwhywhere) - get parent and all childs

ndash kslfre(laddr) - free the latch

Oracle give us possibility to do the same by oradebug call

RDTEX | Latch internals | RUOUG Seminar

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313501

983125983155983141 32983085983138983145983156 983152983151983154983156983155

64983138983145983156 O983154983137983139983148983141 983139983151983154983154983157983152983156983155 983156983144983141 983137983154983143983157983149983141983150983156983155 983151983142 983151983154983137983140983141983138983157983143 983139983137983148983148 983145983150 10204 983137983150983140 983137983138983151983158983141

18

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1982

ldquoWHERErdquo AND ldquoWHYrdquo ORACLE GETS THE LATCH

To request the latch Oracle kernel routines need

bull laddres ndash address of latch in SGAbull wait ndash flag for no-wait (0) or wait (1) latch acquisition

bull where ndash integer code for location from where the latch is acquired

Oracle lists possible latch ldquowhererdquo values in x$ksllwindxbull why - context of why the latch is acquiring at this ldquowhererdquo It

contains DBA for CBC latches (Tanel Poder) SGA chunk addressfor shared pool latch session address etchellip ldquoWhyrdquo meaning for

some ldquowhererdquo can be found in x$ksllwksllwlbl

bull mode ndash requesting state for shared lathes

ndash 8 SHARED mode 16 EXCLUSIVE mode

RDTEX | Latch internals | RUOUG Seminar

19

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2082

LIST OF LATCH ldquoWHERErdquo LOCATIONS

X$KSLLW ndash List of latch [W]here Part of v$latch_misses

ndash ksllwnam ndash Kernel function name(ldquokghupr1rdquo ldquoksqdelrdquo etc) ndash ksllwlbl ndash Description of ldquowhyrdquo context

Join this code on indx to

ndash x$ksuprlatksulawhr (v$latchholder)ndash ldquowhererdquo value that wasused by latch holder

ndash x$kslltkslltwhr (v$latch) ndash ldquowhererdquo that was used by lastsuccessful latch acquisition

ndash x$ksuprksllawer (v$process) ndash ldquowhererdquo that was used byprocess is being waiting for latch

ndash x$kslwscindx (v$latch_misses) ndash latch sleep statistics by

ldquowhererdquo

RDTEX | Latch internals | RUOUG Seminar

20

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2182

V$LATCH_MISSES

bull Based on X$KSLWSC - No[W]ait and [S]leep [C]ount stats byContext Documentation is rather obscure

bull The following counters are incremented on each latch sleep

sleep_count (ldquowhererdquo the latch was held)

wtr_slp_count (ldquowhererdquo the latch get was requested)

bull nwfail_count is incremented only for session which SID is equalto _LATCH_MISS_STAT_SID parameter

bull longhold_count is incremented for shared latches only when

someone held the latch at this ldquowhererdquo for the entire duration ofsleep

RDTEX | Latch internals | RUOUG Seminar

21

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2282

LATCH IS HOLDING BY PROCESS NOT SESSION

bull

RDTEX | Latch internals | RUOUG Seminar

983123983156983154983157983139983156 983147983155983157983152983154 983163

991270

983123983156983154983157983139983156 983147983155983148983148983137983163983147983155983148983148983156 98314798315598314898314898313798314898313798315698313114983133

983165

991270983165

983120983154983151983139983141983155983155 983142983145983160983141983140 983137983154983154983137983161

983158$983152983154983151983139983141983155983155 983085gt 983160$983147983155983157983152983154

983123983156983154983157983139983156 983147983155983148983148983156983163

991270

983165

L983145983155983156 983151983142 983137983148983148 983148983137983156983139983144983141983155

983158$983148983137983156983139983144 983085gt983160$983147983155983148983148983156

Each process has an array of references to the latches it is holding

22

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2382

KSLLA IS HIDDEN IN V$PROCESS

bull The process latching info is the kslla structure embedded in theprocess state object

bull Some kslla fields were externalized to SQL

bull kslla contains array of latches holding by process at each level andinformation about the state of any current attempt to acquire a

latch Not externalized directly

bull When you query v$latchholder x$ksuprlat Oracle scansv$processx$ksupr in order to find all the processes that are

holding the latchesbull Tanel Poder uses high frequency sampling of v$latchholder to

systematically troubleshoot latch contention

RDTEX | Latch internals | RUOUG Seminar

23

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2482

X$KSUPRKSLLA FIELDS INSTRUMENT THE LATCH GET

bull ksllalaq ndash address of latch acquiring Populated during immediateget (and spin before 11g)

bull ksllawat - latch being waited for This is v$processlatchwait

bull ksllawhy ndash ldquowhyrdquo for the latch being waited for

bull ksllawere ndash ldquowhererdquo for the latch being waited for

bull ksllalow ndash bit array of levels of currently holding latches

bull ksllaspn - latch this process is spinning on v$processlatchspinNot populated since 81

bull ksllaps - inter-process post statistics

RDTEX | Latch internals | RUOUG Seminar

24

Generic ldquolatch freerdquo and 27 latch specific ldquolatch free rdquo wait events areassociated with sleep only Spin is invisible to Oracle Wait Interface

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2582

THE LATCH STRUCTURE ndash KSLLT

RDTEX | Latch internals | RUOUG Seminar

struct ksllt

ltLatchgt

ldquowhererdquo and ldquowhyrdquo

Level latch class other attributes

Statistics

Latch wait list header

hellip

25

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2682

LATCH SIZE BY VERSION

nix 32bit nix 64bit Windows 32bit

734 92 - 120

806 104 - 104817 104 144 104

901 200 160

920 196 240 200

1010 256 208

1020 - 11203 100 160 104

RDTEX | Latch internals | RUOUG Seminar

x$ksmfsv ndash list of all fixed SGA variables983123E983116EC983124 DI983123983124I983118C983124 983147983155983149983142983155983155983145983162

F983122983119983117 983160$983147983155983149983142983155983158

983127HE983122E 983147983155983149983142983155983156983161983152 = 983147983155983148983148983156

Latch structure was bigger in 101 due to additional latch statistics

26

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2782

ORACLE LATCH IS NOT JUST A SINGLE MEMORY LOCATION

bull Before 11g Value of first latch byte (word for shared latches) wasused to determine latch state

0x00 ndash latch is free

0xFF ndash exclusive latch is busy Was 0x01 in Oracle 7

0x010x02hellip - shared latch holding by 12 hellip processes simultaneously

0x20000000 | pid - shared latch holding exclusively

bull In 11g first latch word shows the pid of the latch holder

0x00 ndash latch is free0x12 ndash Oracle process with pid 18 holds the exclusive latch

RDTEX | Latch internals | RUOUG Seminar

27

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2882

BUSY ORACLE 11G LATCH IN MEMORY

bull 112 Exclusive latch (32-bit platform)

bull 112 Shared latch EXCLUSIVE mode

RDTEX | Latch internals | RUOUG Seminar

SQLgt oradebug peek 200222A0 24

[200222A0 200222B8) = 00000016 00000001 000001D0 00000007

pid^ gets latch level

SQLgt oradebug peek 0xampladdr 24

[2000C814 2000C82C) = 2000001F 0000007C 0000007E 00000006 pid^ gets latch level

983139983151983150983156 983155983153983148983152983148983157983155 983150983151983148983151983143 9831409831419831499831519831350128

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2982

LATCH FIXED TABLES

bull Before Oracle 11g all the latches were externalized to SQL inv$latch (x$ksllt)

bull In 11g parent (fixed SGA) latches are externalized inv$latch_parent (x$kslltr_parent)

bull In 11g child (allocated on startup) latches are externalized in

v$latch_children (x$kslltr_children) bull Oracle 11g v$latch (x$kslltr) merges these data Beware select

from x$kslltr acquires all parent latches and their children

bull Only part of latch structure is visible in x$ksllthellip fixed tables ndash kslltwhr kslltwhy ndash ldquowhererdquo and ldquowhyrdquo latch was acquired last time

ndash kslltwgt hellip - statistics

ndash kslltlvl hellip - attributes

ndash kslltwtt ndash latch wait time

RDTEX | Latch internals | RUOUG Seminar

29

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3082

CUMULATIVE V$LATCH STATISTIC COUNTERS

RDTEX | Latch internals | RUOUG Seminar

Statistics x$ksllt When and how it increments

GETS kslltwgt ldquo++rdquo after wait mode latch get

MISSES kslltwff ldquo++rdquo after wait get if it was missed

SLEEPS kslltwsl ldquo+number_of_sleepsrdquo during get

SPIN_GETS ksllthst0 ldquo++rdquo if get was missed but not slept

WAIT_TIME kslltwtt ldquo+wait_timerdquo after latch get

IMMEDIATE_GETS kslltngt ldquo++rdquo after nowait mode latch get Is notincremented on miss

IMMEDIATE_MISSES kslltnfa ldquo++rdquo if nowait mode get was missed

30

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3182

MANY STATISTICS WERE LOST IN 102

In 102 Oracle made obsolete many useful latch statistics

Latch size was decreased and the following fields always contain 0now in v$latch

ndash v$latchwaiters_woken

ndash v$latchwaits_holding_latch

ndash v$latchsleep1hellipsleep10

ndash and many undocumented statistics

This was mentioned in 102 Reference

RDTEX | Latch internals | RUOUG Seminar

31

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3282

X$KSLLD - LATCH DESCRIPTORS

bull Flags and attributes describing each latch are stored in kslldt arrayof kslld structures

bull [K]ernel [S]ervice [L]atch [L]ock [D]escriptor

bull Some but not all of kslld fields are externalized via x$kslld andv$latchname

bull Located in the PGA

RDTEX | Latch internals | RUOUG Seminar

32

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382

LATCH ATTRIBUTES

bull Each latch have at least the following attributes in kslldt

ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared

ndash PAR Is the latch Solitary or Parent for the family of child latches

ndash G2C Can two child latches be simultaneously requested in wait mode

using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92

ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above

ndash Level 0-14 Used to prevent latch deadlocks

ndash Class 0-7 Spin and wait class assigned to the latch 92 and above

RDTEX | Latch internals | RUOUG Seminar

33

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482

LATCH ATTRIBUTES BY ORACLE VERSION

RDTEX | Latch internals | RUOUG Seminar

11203 553 154 72 - 6 93

Oracle Number of latches PAR G2C LNG UFS SHARED

7340 53 14 2 3 - -

8063 80 21 7 3 - 38174 152 48 19 4 - 9

9208 242 79 37 - - 19

10202 385 114 55 - 4 47

10203 388 117 58 - 4 48

10204 394 117 59 - 4 50

11106 496 145 67 - 6 81

11107 502 145 67 - 6 83

11201 535 149 70 - 6 86

11202 551 153 72 - 6 91

34

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582

LATCH LEVELS

bull Every latch has a level in the range 0-14 to avoid deadlocks

bull Process can request a latch X in wait mode after obtaining latch Y if and

only if level X gt level Y

bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y

bull If these rules are broken the Oracle process raises

ORA-600[504] - trying to obtain another latch at incompatible level Note 281041

ORA-600[525] - trying to obtain another latch at the same level Note 1388881

ORA-600[526] - latch child order violation Note 1388891

RDTEX | Latch internals | RUOUG Seminar

35

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682

PARENT AND CHILD LATCHES

bull Latch can be a parent child or solitary latch

bull Both parent and child latches share the same latch name

bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss

bull Child latches are allocated during instance startup

bull The parent latch can be gotten independently

bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s

RDTEX | Latch internals | RUOUG Seminar

(latch info) wait_event=0 bits=10

holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl

Context saved from call 0

state=busy

36

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782

LATCH TREES

ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches

Direct SGA access program output for 9206 instance with too smallshared pool

RDTEX | Latch internals | RUOUG Seminar

ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1

waiter ospid 129 sid 72 pid 45

holding a154b7120 library cache (157) level=5 child=17 kglupc child

waiter ospid 18255 sid 65 pid 930

waiter ospid 6690 sid 554 pid 1654

waiter ospid 4685 sid 879 pid 1034

hellip

waiter ospid 29749 sid 180 pid 155

holding a154b7db8 library cache (157) level=5 child=4 kglupc child

waiter ospid 13104 sid 281 pid 220

waiter ospid 24089 sid 565 pid 636

waiter ospid 25002 sid 621 pid 1481

waiter ospid 16930 sid 1046 pid 783hellip

37

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882

LATCH CLASSES

bull Classes with different spin and wait policies

bull Up to 8 classes By default all the latches belong to class 0

bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2

bull Latch can be assigned to class by its number For example

ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000

ndash _LATCH_CLASSES=46

bull Will be discussed later

RDTEX | Latch internals | RUOUG Seminar

SQLgt select from x$ksllclass

INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4

---- ------ ----- -------- ------ ------ ------ ------ ------

0 20000 0 1 8000 8000 8000 8000 8000

hellip

6 100 12 1000 2000 3000 4000 4000 4000

38

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982

LATCH ACQUISITION IN NO-WAIT MODE

bull kslgetl(laddr0whywhere) ndash Exclusive latch get

bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep

ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches

ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches

bull IMMEDIATE_ statistics are updated

bull Used when many candidate latches are available to bypass the

latch level rule or as part of CBC chain examination

RDTEX | Latch internals | RUOUG Seminar

sqlplus nolog demo_03sql39

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082

LATCH WAITS

RDTEX | Latch internals | RUOUG Seminar 40

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182

WAITING FOR THE LATCH

RDTEX | Latch internals | RUOUG Seminar

983123 G A

C983120983125 1 C983120983125 2

983120983154983151983139983141983155983155 A

983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)

983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144

983116983137983156983139983144

983120983154983151983139983141983155983155 B

41

RDTEX | L h i l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282

LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION

bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries

again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of

times

bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep

bull A sleep corresponds to a wait for the latch free wait event

bull Initially a process sleeps for 001 s This time is doubled in every

subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip

RDTEX | Latch internals | RUOUG Seminar

Still did not found any versionwhich used this

May be it was lt= Oracle 6

42

RDTEX | L t h i t l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382

LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81

Latch gets with wait (kslgetl(laddress1hellip))

ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times

ndash Sleep on latch free event with exponential backoff

ndash Repeat

RDTEX | Latch internals | RUOUG Seminar

43

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482

8I LATCH SPIN CODE FLOW USING DTRACE

RDTEX | Latch internals | RUOUG Seminar

bull hellip Event 10046 trace

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2

983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29

983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270

sqlplus nolog demo_04_8sql 29

44

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582

POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF

bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec

RDTEX | Latch internals | RUOUG Seminar

12 ]2 )1[(

minus= +wait N

timeout

45

bull Most waits were for nothing ndash latch already was free

bull Latch utilization can not be high

bull Unnecessary spins provoke CPU thrashing

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682

92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE

sqlplus nolog demo_05sql 45

RDTEX | Latch internals | RUOUG Seminar

kslgetl(0x50006318 1)

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt kslges(0x50006318 ) -wait latch get

-gt skgslsgts(0x50006318 ) -spin latch get

-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT

-gt kskthbwt(0x0)

-gt kslwlmod() - set up Wait List

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt skgpwwait -sleep latch getsemop(11 17-10 1)

983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140

46

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782

RELIABLE LATCH WAITS

bull Hidden latch wait revolution that we missed Latch relies on waitposting

bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout

bull Exclusive latch spins 20000 cycles by default

bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock

bull Latches assigned to non-default class wait until timeout

bull Version 90 was transient and used mix of backoff and wait posting

This is why _LATCH_WAIT_POSTING parameter disappeared in9i

bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin

RDTEX | Latch internals | RUOUG Seminar

47

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882

IMAGINE THE TIME MICROSCOPE

| a a s | a

Reality One million zoomed time

1 us 1 sec

1 ms 17 min1 sec 115 days

Light speed (300000 kmsec) Sonic speed (300 msec)

CPU tick (2 GHz) 00005 sec

Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec

Max spin time for shared latch (~220005 ticks) ~10 sec

Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec

Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec

OS context switch time (10us-1ms) 10sec-17min

48

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982

LATCH WAITS UNDER TIME MICROSCOPE

Old 8i latch algorithmSpin for latch during 5 seconds

Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min

hellipSpin again for 5 seconds

Sleep again for 23 days (2 sec) hellip

New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)

If spin not succeeded go to sleep until post

According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)

| |

49

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082

_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES

bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and

_LATCH_CLASS_0 parameters

bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000

bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter

bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch

50

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182

TIMED LATCH WAIT POSTING

bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel

bull This can lead to instance hang ndash process will never be woken up

bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout

bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem

bull Generic AIX ports always use timed latch wait posting

bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux

51

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282

FINE GRAIN LATCH TUNING

bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting

bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7

bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo

microseconds using pollsys() (not semtimedop()) system call

ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times

Loop up to ldquoSpinsrdquo cycles

Yield CPU using yield() (or sched_yield())

ndash Sleep for ldquoSleepXrdquo usecs

ndash Then spin again hellip

52

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382

TRACES OF NONDEFAULT CLASS LATCH

983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152

991270

9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000

991270 E983158983141983150983156 10046 983156983154983137983139983141

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482

PERFORMANCE OF LATCH CLASSES

Row cache objects latch contention testcase on ODA (24 SMT)54

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582

IS THE _SPIN_COUNT REALLY STATIC

bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch

waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used

ldquocache buffers chainrdquo latch contention

bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass

(_LATCH_CLASS_X) By default ndash 20000

bull Shared latch spin can be tuned by _SPIN_COUNT value By

default ndash 2000 However this is not trivial

55

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682

SHARED LATCH BEHAVE LIKE ENQUEUE

bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80

bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests

bull This blocking achieved by special 0x40000000 bit in the latch

value This bit is a flag that some incompatible latch acquisitionis in progress

bull Queued processes will be woken up one by one on latch release

bull X mode latch gets effectively serialize the shared latch

bull See my blog for description of experiments

56

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782

TRACE OF SHARED LATCH WAIT

Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the

latch wait list and then spins again

983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141

983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156

983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983147983155983148983159983148983149983151983140()

983147983155983148983155983147983143983155(098316050009BAC )

983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140

983155983141983149983151983152(27)

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882

SHARED LATCH ACQUISITION

bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned

bull X mode shared latch get spins by default up to 4000 cycles

bull S mode does not spin at all (or spins in unknown way)

S mode get X mode get

Held in S mode Compatible 2_SPIN_COUNT

Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT

58

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982

LATCH RELEASE

bull Free the latch ndash kslfre(laddr)

bull Oracle process releases the latch nonatomically

bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process

bull This requires less bus invalidation and ensures propagation of latch

release to other local cachesbull Not fair policy Spinners on the local CPU board have the

preference

bull Then process posts the first process in the list of waiters

59

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082

LATCH CONTENTION

RDTEX | Latch internals | RUOUG Seminar60

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182

LATCH CONTENTION

bull Occurs when a latch is required by several processes at thesame time

bull Mostly due to bugs or abnormal application behavior

bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp

bull To diagnose the latch contention we need to investigate thelatch statistics [8]

61

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282

DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS

Latch requests arrival rate

Immediate gets efficiency

Latch sleeps ratio

Latch wait time per second

Latch spin efficiencymisses

getsspin∆

∆= _σ

misses

sleepsK

∆=

time

gets

∆=λ

gets

misses

∆= ρ

time

timewait W

∆=

_

Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics

62

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382

SAMPLED AVERAGES

By sampling of x$ksupr we can estimate

ndash Length of latch wait queue

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS

ndash Number of spinning processes in 10g

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS

Sampling of v$lathholder estimates latch utilization

Using DTrace we can measure

ndash Latch acquisition and holding times

ndash Actual spin count

w L

s N

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482

DERIVED LATCH STATISTICS

Latch utilization (PASTA)

Average holding time

Length of latch wait list

Recurrent sleeps ratio

Latch acquisition time

time

timeholdinglatchU

∆=asymp

__ ρ

Re_100

___

questsGet

TimeSnap MissGet Pct S

lowast==

λ

W L =

κ

κ σ 1minus+

)(1

W N T saq += minusλ

64

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 10: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1082

THE NEED FOR SPIN

RDTEX | Latch internals | RUOUG Seminar

10

Time to complete CBC latch contention testcase vs number of threads

No spinning _spin_count=0

_spin_count=2000

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1182

SPINLOCK REALIZATIONS

RDTEX | Latch internals | RUOUG Seminar

Spinlock Pseudocode Problems

TS

pre-112 mutex

while(Test_and_Set(lock)) Bus saturation

TTS

Oracle latch

while(lock||Test_and_Set(lock)) Invalidation storms onrelease (ldquoopen doorrdquo

ldquothundering herdsrdquo)

Delay

Latch and 112

Mutex

Adjustable delay Slower under contention

Anderson MCSetc

Queues Widely used in JavaLinux kernel hellip

CPU and memoryoverhead preemptionissues

11

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1282

THE TOOLS

RDTEX | Latch internals | RUOUG Seminar12

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1382

DTRACE SOLARIS DYNAMIC TRACING FRAMEWORK

DTrace allows us to investigate how Oracle latches perform in real time ndash Create triggers on any event inside Solaris and function call inside Oracle

providermodulefunctionname

pid1910oraclekslgetlentry ndash Write trigger bodies ndash actions May read and change any address location ndash Can count the latch spins trace the latch waits perform experiments ndash Measure times and distributions up to microseconds precision

RDTEX | Latch internals | RUOUG Seminar

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313500

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1482

ORADEBUG INTERNAL ORACLE DEBUGGER

RDTEX | Latch internals | RUOUG Seminar

bull oradebug call Allows us invoke any internal Oracle function manually

SQLgt oradebug peek 200222A0 24

[200222A0 200222B8) = 00000016 00000001 000001D0 00000007

SQLgt oradebug call kslgetl 0x5001A020 1 2 3Function returned 1

bull oradebug peek Examines contents of any memory location

bull oradebug poke Modifies memory

bull oradebug watch Watch a region of memory

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1582

LATCH CONTENTION TESTCASES

RDTEX | Latch internals | RUOUG Seminar

15

create table latch_contention as select from dualhellipfor i in 11000000loopselect dummy into a from latch_contention

end loop

bull Exclusive and shared latches behave differentlybull Contention for shared cache buffer chains latch arises when manysessions scan the same block concurrently and _fastpin_enable=0

bull To induce the contention for exclusive ldquorow cache objectsrdquo latch protectingdc_users row cache just add several dynamic RLS policies

exec dbms_rlsadd_policy(systemlatch_contentionrls_1 hellip

exec dbms_rlsadd_policy(systemlatch_contentionrls_2 helliphellip

bull These dummy policies generate predicates 1=1bull Many other contention scenarios possible

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313509831351

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1682

ROW CACHE LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

16

Elapsed and CPU times vs number of threadsOracle Database Appliance (24 SMT)

Latch wait

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1782

HOW ORACLEREQUESTS THE LATCH

RDTEX | Latch internals | RUOUG Seminar17

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1882

DTRACE REVEALS LATCH INTERFACE ROUTINES

Oracle calls the following functions to acquire the latch

ndash kslgetl(laddr wait why where) - get exclusive latch ndash kslg2c(l1l2trcwhy where) - get two excl child latches

ndash kslgetsl (laddrwaitwhywheremode) - get shared latch

bull 11g - ksl_get_shared_latch (hellip)

ndash kslg2cs(l1l2modetrcwhy where)) - get two shr child latches

ndash kslgpl(laddrcommentwhywhere) - get parent and all childs

ndash kslfre(laddr) - free the latch

Oracle give us possibility to do the same by oradebug call

RDTEX | Latch internals | RUOUG Seminar

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313501

983125983155983141 32983085983138983145983156 983152983151983154983156983155

64983138983145983156 O983154983137983139983148983141 983139983151983154983154983157983152983156983155 983156983144983141 983137983154983143983157983149983141983150983156983155 983151983142 983151983154983137983140983141983138983157983143 983139983137983148983148 983145983150 10204 983137983150983140 983137983138983151983158983141

18

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1982

ldquoWHERErdquo AND ldquoWHYrdquo ORACLE GETS THE LATCH

To request the latch Oracle kernel routines need

bull laddres ndash address of latch in SGAbull wait ndash flag for no-wait (0) or wait (1) latch acquisition

bull where ndash integer code for location from where the latch is acquired

Oracle lists possible latch ldquowhererdquo values in x$ksllwindxbull why - context of why the latch is acquiring at this ldquowhererdquo It

contains DBA for CBC latches (Tanel Poder) SGA chunk addressfor shared pool latch session address etchellip ldquoWhyrdquo meaning for

some ldquowhererdquo can be found in x$ksllwksllwlbl

bull mode ndash requesting state for shared lathes

ndash 8 SHARED mode 16 EXCLUSIVE mode

RDTEX | Latch internals | RUOUG Seminar

19

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2082

LIST OF LATCH ldquoWHERErdquo LOCATIONS

X$KSLLW ndash List of latch [W]here Part of v$latch_misses

ndash ksllwnam ndash Kernel function name(ldquokghupr1rdquo ldquoksqdelrdquo etc) ndash ksllwlbl ndash Description of ldquowhyrdquo context

Join this code on indx to

ndash x$ksuprlatksulawhr (v$latchholder)ndash ldquowhererdquo value that wasused by latch holder

ndash x$kslltkslltwhr (v$latch) ndash ldquowhererdquo that was used by lastsuccessful latch acquisition

ndash x$ksuprksllawer (v$process) ndash ldquowhererdquo that was used byprocess is being waiting for latch

ndash x$kslwscindx (v$latch_misses) ndash latch sleep statistics by

ldquowhererdquo

RDTEX | Latch internals | RUOUG Seminar

20

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2182

V$LATCH_MISSES

bull Based on X$KSLWSC - No[W]ait and [S]leep [C]ount stats byContext Documentation is rather obscure

bull The following counters are incremented on each latch sleep

sleep_count (ldquowhererdquo the latch was held)

wtr_slp_count (ldquowhererdquo the latch get was requested)

bull nwfail_count is incremented only for session which SID is equalto _LATCH_MISS_STAT_SID parameter

bull longhold_count is incremented for shared latches only when

someone held the latch at this ldquowhererdquo for the entire duration ofsleep

RDTEX | Latch internals | RUOUG Seminar

21

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2282

LATCH IS HOLDING BY PROCESS NOT SESSION

bull

RDTEX | Latch internals | RUOUG Seminar

983123983156983154983157983139983156 983147983155983157983152983154 983163

991270

983123983156983154983157983139983156 983147983155983148983148983137983163983147983155983148983148983156 98314798315598314898314898313798314898313798315698313114983133

983165

991270983165

983120983154983151983139983141983155983155 983142983145983160983141983140 983137983154983154983137983161

983158$983152983154983151983139983141983155983155 983085gt 983160$983147983155983157983152983154

983123983156983154983157983139983156 983147983155983148983148983156983163

991270

983165

L983145983155983156 983151983142 983137983148983148 983148983137983156983139983144983141983155

983158$983148983137983156983139983144 983085gt983160$983147983155983148983148983156

Each process has an array of references to the latches it is holding

22

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2382

KSLLA IS HIDDEN IN V$PROCESS

bull The process latching info is the kslla structure embedded in theprocess state object

bull Some kslla fields were externalized to SQL

bull kslla contains array of latches holding by process at each level andinformation about the state of any current attempt to acquire a

latch Not externalized directly

bull When you query v$latchholder x$ksuprlat Oracle scansv$processx$ksupr in order to find all the processes that are

holding the latchesbull Tanel Poder uses high frequency sampling of v$latchholder to

systematically troubleshoot latch contention

RDTEX | Latch internals | RUOUG Seminar

23

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2482

X$KSUPRKSLLA FIELDS INSTRUMENT THE LATCH GET

bull ksllalaq ndash address of latch acquiring Populated during immediateget (and spin before 11g)

bull ksllawat - latch being waited for This is v$processlatchwait

bull ksllawhy ndash ldquowhyrdquo for the latch being waited for

bull ksllawere ndash ldquowhererdquo for the latch being waited for

bull ksllalow ndash bit array of levels of currently holding latches

bull ksllaspn - latch this process is spinning on v$processlatchspinNot populated since 81

bull ksllaps - inter-process post statistics

RDTEX | Latch internals | RUOUG Seminar

24

Generic ldquolatch freerdquo and 27 latch specific ldquolatch free rdquo wait events areassociated with sleep only Spin is invisible to Oracle Wait Interface

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2582

THE LATCH STRUCTURE ndash KSLLT

RDTEX | Latch internals | RUOUG Seminar

struct ksllt

ltLatchgt

ldquowhererdquo and ldquowhyrdquo

Level latch class other attributes

Statistics

Latch wait list header

hellip

25

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2682

LATCH SIZE BY VERSION

nix 32bit nix 64bit Windows 32bit

734 92 - 120

806 104 - 104817 104 144 104

901 200 160

920 196 240 200

1010 256 208

1020 - 11203 100 160 104

RDTEX | Latch internals | RUOUG Seminar

x$ksmfsv ndash list of all fixed SGA variables983123E983116EC983124 DI983123983124I983118C983124 983147983155983149983142983155983155983145983162

F983122983119983117 983160$983147983155983149983142983155983158

983127HE983122E 983147983155983149983142983155983156983161983152 = 983147983155983148983148983156

Latch structure was bigger in 101 due to additional latch statistics

26

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2782

ORACLE LATCH IS NOT JUST A SINGLE MEMORY LOCATION

bull Before 11g Value of first latch byte (word for shared latches) wasused to determine latch state

0x00 ndash latch is free

0xFF ndash exclusive latch is busy Was 0x01 in Oracle 7

0x010x02hellip - shared latch holding by 12 hellip processes simultaneously

0x20000000 | pid - shared latch holding exclusively

bull In 11g first latch word shows the pid of the latch holder

0x00 ndash latch is free0x12 ndash Oracle process with pid 18 holds the exclusive latch

RDTEX | Latch internals | RUOUG Seminar

27

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2882

BUSY ORACLE 11G LATCH IN MEMORY

bull 112 Exclusive latch (32-bit platform)

bull 112 Shared latch EXCLUSIVE mode

RDTEX | Latch internals | RUOUG Seminar

SQLgt oradebug peek 200222A0 24

[200222A0 200222B8) = 00000016 00000001 000001D0 00000007

pid^ gets latch level

SQLgt oradebug peek 0xampladdr 24

[2000C814 2000C82C) = 2000001F 0000007C 0000007E 00000006 pid^ gets latch level

983139983151983150983156 983155983153983148983152983148983157983155 983150983151983148983151983143 9831409831419831499831519831350128

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2982

LATCH FIXED TABLES

bull Before Oracle 11g all the latches were externalized to SQL inv$latch (x$ksllt)

bull In 11g parent (fixed SGA) latches are externalized inv$latch_parent (x$kslltr_parent)

bull In 11g child (allocated on startup) latches are externalized in

v$latch_children (x$kslltr_children) bull Oracle 11g v$latch (x$kslltr) merges these data Beware select

from x$kslltr acquires all parent latches and their children

bull Only part of latch structure is visible in x$ksllthellip fixed tables ndash kslltwhr kslltwhy ndash ldquowhererdquo and ldquowhyrdquo latch was acquired last time

ndash kslltwgt hellip - statistics

ndash kslltlvl hellip - attributes

ndash kslltwtt ndash latch wait time

RDTEX | Latch internals | RUOUG Seminar

29

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3082

CUMULATIVE V$LATCH STATISTIC COUNTERS

RDTEX | Latch internals | RUOUG Seminar

Statistics x$ksllt When and how it increments

GETS kslltwgt ldquo++rdquo after wait mode latch get

MISSES kslltwff ldquo++rdquo after wait get if it was missed

SLEEPS kslltwsl ldquo+number_of_sleepsrdquo during get

SPIN_GETS ksllthst0 ldquo++rdquo if get was missed but not slept

WAIT_TIME kslltwtt ldquo+wait_timerdquo after latch get

IMMEDIATE_GETS kslltngt ldquo++rdquo after nowait mode latch get Is notincremented on miss

IMMEDIATE_MISSES kslltnfa ldquo++rdquo if nowait mode get was missed

30

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3182

MANY STATISTICS WERE LOST IN 102

In 102 Oracle made obsolete many useful latch statistics

Latch size was decreased and the following fields always contain 0now in v$latch

ndash v$latchwaiters_woken

ndash v$latchwaits_holding_latch

ndash v$latchsleep1hellipsleep10

ndash and many undocumented statistics

This was mentioned in 102 Reference

RDTEX | Latch internals | RUOUG Seminar

31

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3282

X$KSLLD - LATCH DESCRIPTORS

bull Flags and attributes describing each latch are stored in kslldt arrayof kslld structures

bull [K]ernel [S]ervice [L]atch [L]ock [D]escriptor

bull Some but not all of kslld fields are externalized via x$kslld andv$latchname

bull Located in the PGA

RDTEX | Latch internals | RUOUG Seminar

32

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382

LATCH ATTRIBUTES

bull Each latch have at least the following attributes in kslldt

ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared

ndash PAR Is the latch Solitary or Parent for the family of child latches

ndash G2C Can two child latches be simultaneously requested in wait mode

using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92

ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above

ndash Level 0-14 Used to prevent latch deadlocks

ndash Class 0-7 Spin and wait class assigned to the latch 92 and above

RDTEX | Latch internals | RUOUG Seminar

33

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482

LATCH ATTRIBUTES BY ORACLE VERSION

RDTEX | Latch internals | RUOUG Seminar

11203 553 154 72 - 6 93

Oracle Number of latches PAR G2C LNG UFS SHARED

7340 53 14 2 3 - -

8063 80 21 7 3 - 38174 152 48 19 4 - 9

9208 242 79 37 - - 19

10202 385 114 55 - 4 47

10203 388 117 58 - 4 48

10204 394 117 59 - 4 50

11106 496 145 67 - 6 81

11107 502 145 67 - 6 83

11201 535 149 70 - 6 86

11202 551 153 72 - 6 91

34

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582

LATCH LEVELS

bull Every latch has a level in the range 0-14 to avoid deadlocks

bull Process can request a latch X in wait mode after obtaining latch Y if and

only if level X gt level Y

bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y

bull If these rules are broken the Oracle process raises

ORA-600[504] - trying to obtain another latch at incompatible level Note 281041

ORA-600[525] - trying to obtain another latch at the same level Note 1388881

ORA-600[526] - latch child order violation Note 1388891

RDTEX | Latch internals | RUOUG Seminar

35

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682

PARENT AND CHILD LATCHES

bull Latch can be a parent child or solitary latch

bull Both parent and child latches share the same latch name

bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss

bull Child latches are allocated during instance startup

bull The parent latch can be gotten independently

bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s

RDTEX | Latch internals | RUOUG Seminar

(latch info) wait_event=0 bits=10

holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl

Context saved from call 0

state=busy

36

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782

LATCH TREES

ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches

Direct SGA access program output for 9206 instance with too smallshared pool

RDTEX | Latch internals | RUOUG Seminar

ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1

waiter ospid 129 sid 72 pid 45

holding a154b7120 library cache (157) level=5 child=17 kglupc child

waiter ospid 18255 sid 65 pid 930

waiter ospid 6690 sid 554 pid 1654

waiter ospid 4685 sid 879 pid 1034

hellip

waiter ospid 29749 sid 180 pid 155

holding a154b7db8 library cache (157) level=5 child=4 kglupc child

waiter ospid 13104 sid 281 pid 220

waiter ospid 24089 sid 565 pid 636

waiter ospid 25002 sid 621 pid 1481

waiter ospid 16930 sid 1046 pid 783hellip

37

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882

LATCH CLASSES

bull Classes with different spin and wait policies

bull Up to 8 classes By default all the latches belong to class 0

bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2

bull Latch can be assigned to class by its number For example

ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000

ndash _LATCH_CLASSES=46

bull Will be discussed later

RDTEX | Latch internals | RUOUG Seminar

SQLgt select from x$ksllclass

INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4

---- ------ ----- -------- ------ ------ ------ ------ ------

0 20000 0 1 8000 8000 8000 8000 8000

hellip

6 100 12 1000 2000 3000 4000 4000 4000

38

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982

LATCH ACQUISITION IN NO-WAIT MODE

bull kslgetl(laddr0whywhere) ndash Exclusive latch get

bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep

ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches

ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches

bull IMMEDIATE_ statistics are updated

bull Used when many candidate latches are available to bypass the

latch level rule or as part of CBC chain examination

RDTEX | Latch internals | RUOUG Seminar

sqlplus nolog demo_03sql39

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082

LATCH WAITS

RDTEX | Latch internals | RUOUG Seminar 40

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182

WAITING FOR THE LATCH

RDTEX | Latch internals | RUOUG Seminar

983123 G A

C983120983125 1 C983120983125 2

983120983154983151983139983141983155983155 A

983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)

983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144

983116983137983156983139983144

983120983154983151983139983141983155983155 B

41

RDTEX | L h i l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282

LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION

bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries

again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of

times

bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep

bull A sleep corresponds to a wait for the latch free wait event

bull Initially a process sleeps for 001 s This time is doubled in every

subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip

RDTEX | Latch internals | RUOUG Seminar

Still did not found any versionwhich used this

May be it was lt= Oracle 6

42

RDTEX | L t h i t l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382

LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81

Latch gets with wait (kslgetl(laddress1hellip))

ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times

ndash Sleep on latch free event with exponential backoff

ndash Repeat

RDTEX | Latch internals | RUOUG Seminar

43

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482

8I LATCH SPIN CODE FLOW USING DTRACE

RDTEX | Latch internals | RUOUG Seminar

bull hellip Event 10046 trace

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2

983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29

983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270

sqlplus nolog demo_04_8sql 29

44

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582

POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF

bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec

RDTEX | Latch internals | RUOUG Seminar

12 ]2 )1[(

minus= +wait N

timeout

45

bull Most waits were for nothing ndash latch already was free

bull Latch utilization can not be high

bull Unnecessary spins provoke CPU thrashing

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682

92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE

sqlplus nolog demo_05sql 45

RDTEX | Latch internals | RUOUG Seminar

kslgetl(0x50006318 1)

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt kslges(0x50006318 ) -wait latch get

-gt skgslsgts(0x50006318 ) -spin latch get

-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT

-gt kskthbwt(0x0)

-gt kslwlmod() - set up Wait List

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt skgpwwait -sleep latch getsemop(11 17-10 1)

983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140

46

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782

RELIABLE LATCH WAITS

bull Hidden latch wait revolution that we missed Latch relies on waitposting

bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout

bull Exclusive latch spins 20000 cycles by default

bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock

bull Latches assigned to non-default class wait until timeout

bull Version 90 was transient and used mix of backoff and wait posting

This is why _LATCH_WAIT_POSTING parameter disappeared in9i

bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin

RDTEX | Latch internals | RUOUG Seminar

47

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882

IMAGINE THE TIME MICROSCOPE

| a a s | a

Reality One million zoomed time

1 us 1 sec

1 ms 17 min1 sec 115 days

Light speed (300000 kmsec) Sonic speed (300 msec)

CPU tick (2 GHz) 00005 sec

Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec

Max spin time for shared latch (~220005 ticks) ~10 sec

Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec

Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec

OS context switch time (10us-1ms) 10sec-17min

48

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982

LATCH WAITS UNDER TIME MICROSCOPE

Old 8i latch algorithmSpin for latch during 5 seconds

Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min

hellipSpin again for 5 seconds

Sleep again for 23 days (2 sec) hellip

New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)

If spin not succeeded go to sleep until post

According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)

| |

49

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082

_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES

bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and

_LATCH_CLASS_0 parameters

bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000

bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter

bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch

50

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182

TIMED LATCH WAIT POSTING

bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel

bull This can lead to instance hang ndash process will never be woken up

bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout

bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem

bull Generic AIX ports always use timed latch wait posting

bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux

51

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282

FINE GRAIN LATCH TUNING

bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting

bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7

bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo

microseconds using pollsys() (not semtimedop()) system call

ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times

Loop up to ldquoSpinsrdquo cycles

Yield CPU using yield() (or sched_yield())

ndash Sleep for ldquoSleepXrdquo usecs

ndash Then spin again hellip

52

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382

TRACES OF NONDEFAULT CLASS LATCH

983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152

991270

9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000

991270 E983158983141983150983156 10046 983156983154983137983139983141

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482

PERFORMANCE OF LATCH CLASSES

Row cache objects latch contention testcase on ODA (24 SMT)54

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582

IS THE _SPIN_COUNT REALLY STATIC

bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch

waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used

ldquocache buffers chainrdquo latch contention

bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass

(_LATCH_CLASS_X) By default ndash 20000

bull Shared latch spin can be tuned by _SPIN_COUNT value By

default ndash 2000 However this is not trivial

55

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682

SHARED LATCH BEHAVE LIKE ENQUEUE

bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80

bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests

bull This blocking achieved by special 0x40000000 bit in the latch

value This bit is a flag that some incompatible latch acquisitionis in progress

bull Queued processes will be woken up one by one on latch release

bull X mode latch gets effectively serialize the shared latch

bull See my blog for description of experiments

56

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782

TRACE OF SHARED LATCH WAIT

Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the

latch wait list and then spins again

983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141

983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156

983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983147983155983148983159983148983149983151983140()

983147983155983148983155983147983143983155(098316050009BAC )

983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140

983155983141983149983151983152(27)

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882

SHARED LATCH ACQUISITION

bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned

bull X mode shared latch get spins by default up to 4000 cycles

bull S mode does not spin at all (or spins in unknown way)

S mode get X mode get

Held in S mode Compatible 2_SPIN_COUNT

Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT

58

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982

LATCH RELEASE

bull Free the latch ndash kslfre(laddr)

bull Oracle process releases the latch nonatomically

bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process

bull This requires less bus invalidation and ensures propagation of latch

release to other local cachesbull Not fair policy Spinners on the local CPU board have the

preference

bull Then process posts the first process in the list of waiters

59

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082

LATCH CONTENTION

RDTEX | Latch internals | RUOUG Seminar60

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182

LATCH CONTENTION

bull Occurs when a latch is required by several processes at thesame time

bull Mostly due to bugs or abnormal application behavior

bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp

bull To diagnose the latch contention we need to investigate thelatch statistics [8]

61

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282

DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS

Latch requests arrival rate

Immediate gets efficiency

Latch sleeps ratio

Latch wait time per second

Latch spin efficiencymisses

getsspin∆

∆= _σ

misses

sleepsK

∆=

time

gets

∆=λ

gets

misses

∆= ρ

time

timewait W

∆=

_

Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics

62

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382

SAMPLED AVERAGES

By sampling of x$ksupr we can estimate

ndash Length of latch wait queue

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS

ndash Number of spinning processes in 10g

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS

Sampling of v$lathholder estimates latch utilization

Using DTrace we can measure

ndash Latch acquisition and holding times

ndash Actual spin count

w L

s N

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482

DERIVED LATCH STATISTICS

Latch utilization (PASTA)

Average holding time

Length of latch wait list

Recurrent sleeps ratio

Latch acquisition time

time

timeholdinglatchU

∆=asymp

__ ρ

Re_100

___

questsGet

TimeSnap MissGet Pct S

lowast==

λ

W L =

κ

κ σ 1minus+

)(1

W N T saq += minusλ

64

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 11: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1182

SPINLOCK REALIZATIONS

RDTEX | Latch internals | RUOUG Seminar

Spinlock Pseudocode Problems

TS

pre-112 mutex

while(Test_and_Set(lock)) Bus saturation

TTS

Oracle latch

while(lock||Test_and_Set(lock)) Invalidation storms onrelease (ldquoopen doorrdquo

ldquothundering herdsrdquo)

Delay

Latch and 112

Mutex

Adjustable delay Slower under contention

Anderson MCSetc

Queues Widely used in JavaLinux kernel hellip

CPU and memoryoverhead preemptionissues

11

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1282

THE TOOLS

RDTEX | Latch internals | RUOUG Seminar12

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1382

DTRACE SOLARIS DYNAMIC TRACING FRAMEWORK

DTrace allows us to investigate how Oracle latches perform in real time ndash Create triggers on any event inside Solaris and function call inside Oracle

providermodulefunctionname

pid1910oraclekslgetlentry ndash Write trigger bodies ndash actions May read and change any address location ndash Can count the latch spins trace the latch waits perform experiments ndash Measure times and distributions up to microseconds precision

RDTEX | Latch internals | RUOUG Seminar

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313500

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1482

ORADEBUG INTERNAL ORACLE DEBUGGER

RDTEX | Latch internals | RUOUG Seminar

bull oradebug call Allows us invoke any internal Oracle function manually

SQLgt oradebug peek 200222A0 24

[200222A0 200222B8) = 00000016 00000001 000001D0 00000007

SQLgt oradebug call kslgetl 0x5001A020 1 2 3Function returned 1

bull oradebug peek Examines contents of any memory location

bull oradebug poke Modifies memory

bull oradebug watch Watch a region of memory

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1582

LATCH CONTENTION TESTCASES

RDTEX | Latch internals | RUOUG Seminar

15

create table latch_contention as select from dualhellipfor i in 11000000loopselect dummy into a from latch_contention

end loop

bull Exclusive and shared latches behave differentlybull Contention for shared cache buffer chains latch arises when manysessions scan the same block concurrently and _fastpin_enable=0

bull To induce the contention for exclusive ldquorow cache objectsrdquo latch protectingdc_users row cache just add several dynamic RLS policies

exec dbms_rlsadd_policy(systemlatch_contentionrls_1 hellip

exec dbms_rlsadd_policy(systemlatch_contentionrls_2 helliphellip

bull These dummy policies generate predicates 1=1bull Many other contention scenarios possible

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313509831351

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1682

ROW CACHE LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

16

Elapsed and CPU times vs number of threadsOracle Database Appliance (24 SMT)

Latch wait

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1782

HOW ORACLEREQUESTS THE LATCH

RDTEX | Latch internals | RUOUG Seminar17

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1882

DTRACE REVEALS LATCH INTERFACE ROUTINES

Oracle calls the following functions to acquire the latch

ndash kslgetl(laddr wait why where) - get exclusive latch ndash kslg2c(l1l2trcwhy where) - get two excl child latches

ndash kslgetsl (laddrwaitwhywheremode) - get shared latch

bull 11g - ksl_get_shared_latch (hellip)

ndash kslg2cs(l1l2modetrcwhy where)) - get two shr child latches

ndash kslgpl(laddrcommentwhywhere) - get parent and all childs

ndash kslfre(laddr) - free the latch

Oracle give us possibility to do the same by oradebug call

RDTEX | Latch internals | RUOUG Seminar

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313501

983125983155983141 32983085983138983145983156 983152983151983154983156983155

64983138983145983156 O983154983137983139983148983141 983139983151983154983154983157983152983156983155 983156983144983141 983137983154983143983157983149983141983150983156983155 983151983142 983151983154983137983140983141983138983157983143 983139983137983148983148 983145983150 10204 983137983150983140 983137983138983151983158983141

18

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1982

ldquoWHERErdquo AND ldquoWHYrdquo ORACLE GETS THE LATCH

To request the latch Oracle kernel routines need

bull laddres ndash address of latch in SGAbull wait ndash flag for no-wait (0) or wait (1) latch acquisition

bull where ndash integer code for location from where the latch is acquired

Oracle lists possible latch ldquowhererdquo values in x$ksllwindxbull why - context of why the latch is acquiring at this ldquowhererdquo It

contains DBA for CBC latches (Tanel Poder) SGA chunk addressfor shared pool latch session address etchellip ldquoWhyrdquo meaning for

some ldquowhererdquo can be found in x$ksllwksllwlbl

bull mode ndash requesting state for shared lathes

ndash 8 SHARED mode 16 EXCLUSIVE mode

RDTEX | Latch internals | RUOUG Seminar

19

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2082

LIST OF LATCH ldquoWHERErdquo LOCATIONS

X$KSLLW ndash List of latch [W]here Part of v$latch_misses

ndash ksllwnam ndash Kernel function name(ldquokghupr1rdquo ldquoksqdelrdquo etc) ndash ksllwlbl ndash Description of ldquowhyrdquo context

Join this code on indx to

ndash x$ksuprlatksulawhr (v$latchholder)ndash ldquowhererdquo value that wasused by latch holder

ndash x$kslltkslltwhr (v$latch) ndash ldquowhererdquo that was used by lastsuccessful latch acquisition

ndash x$ksuprksllawer (v$process) ndash ldquowhererdquo that was used byprocess is being waiting for latch

ndash x$kslwscindx (v$latch_misses) ndash latch sleep statistics by

ldquowhererdquo

RDTEX | Latch internals | RUOUG Seminar

20

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2182

V$LATCH_MISSES

bull Based on X$KSLWSC - No[W]ait and [S]leep [C]ount stats byContext Documentation is rather obscure

bull The following counters are incremented on each latch sleep

sleep_count (ldquowhererdquo the latch was held)

wtr_slp_count (ldquowhererdquo the latch get was requested)

bull nwfail_count is incremented only for session which SID is equalto _LATCH_MISS_STAT_SID parameter

bull longhold_count is incremented for shared latches only when

someone held the latch at this ldquowhererdquo for the entire duration ofsleep

RDTEX | Latch internals | RUOUG Seminar

21

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2282

LATCH IS HOLDING BY PROCESS NOT SESSION

bull

RDTEX | Latch internals | RUOUG Seminar

983123983156983154983157983139983156 983147983155983157983152983154 983163

991270

983123983156983154983157983139983156 983147983155983148983148983137983163983147983155983148983148983156 98314798315598314898314898313798314898313798315698313114983133

983165

991270983165

983120983154983151983139983141983155983155 983142983145983160983141983140 983137983154983154983137983161

983158$983152983154983151983139983141983155983155 983085gt 983160$983147983155983157983152983154

983123983156983154983157983139983156 983147983155983148983148983156983163

991270

983165

L983145983155983156 983151983142 983137983148983148 983148983137983156983139983144983141983155

983158$983148983137983156983139983144 983085gt983160$983147983155983148983148983156

Each process has an array of references to the latches it is holding

22

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2382

KSLLA IS HIDDEN IN V$PROCESS

bull The process latching info is the kslla structure embedded in theprocess state object

bull Some kslla fields were externalized to SQL

bull kslla contains array of latches holding by process at each level andinformation about the state of any current attempt to acquire a

latch Not externalized directly

bull When you query v$latchholder x$ksuprlat Oracle scansv$processx$ksupr in order to find all the processes that are

holding the latchesbull Tanel Poder uses high frequency sampling of v$latchholder to

systematically troubleshoot latch contention

RDTEX | Latch internals | RUOUG Seminar

23

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2482

X$KSUPRKSLLA FIELDS INSTRUMENT THE LATCH GET

bull ksllalaq ndash address of latch acquiring Populated during immediateget (and spin before 11g)

bull ksllawat - latch being waited for This is v$processlatchwait

bull ksllawhy ndash ldquowhyrdquo for the latch being waited for

bull ksllawere ndash ldquowhererdquo for the latch being waited for

bull ksllalow ndash bit array of levels of currently holding latches

bull ksllaspn - latch this process is spinning on v$processlatchspinNot populated since 81

bull ksllaps - inter-process post statistics

RDTEX | Latch internals | RUOUG Seminar

24

Generic ldquolatch freerdquo and 27 latch specific ldquolatch free rdquo wait events areassociated with sleep only Spin is invisible to Oracle Wait Interface

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2582

THE LATCH STRUCTURE ndash KSLLT

RDTEX | Latch internals | RUOUG Seminar

struct ksllt

ltLatchgt

ldquowhererdquo and ldquowhyrdquo

Level latch class other attributes

Statistics

Latch wait list header

hellip

25

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2682

LATCH SIZE BY VERSION

nix 32bit nix 64bit Windows 32bit

734 92 - 120

806 104 - 104817 104 144 104

901 200 160

920 196 240 200

1010 256 208

1020 - 11203 100 160 104

RDTEX | Latch internals | RUOUG Seminar

x$ksmfsv ndash list of all fixed SGA variables983123E983116EC983124 DI983123983124I983118C983124 983147983155983149983142983155983155983145983162

F983122983119983117 983160$983147983155983149983142983155983158

983127HE983122E 983147983155983149983142983155983156983161983152 = 983147983155983148983148983156

Latch structure was bigger in 101 due to additional latch statistics

26

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2782

ORACLE LATCH IS NOT JUST A SINGLE MEMORY LOCATION

bull Before 11g Value of first latch byte (word for shared latches) wasused to determine latch state

0x00 ndash latch is free

0xFF ndash exclusive latch is busy Was 0x01 in Oracle 7

0x010x02hellip - shared latch holding by 12 hellip processes simultaneously

0x20000000 | pid - shared latch holding exclusively

bull In 11g first latch word shows the pid of the latch holder

0x00 ndash latch is free0x12 ndash Oracle process with pid 18 holds the exclusive latch

RDTEX | Latch internals | RUOUG Seminar

27

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2882

BUSY ORACLE 11G LATCH IN MEMORY

bull 112 Exclusive latch (32-bit platform)

bull 112 Shared latch EXCLUSIVE mode

RDTEX | Latch internals | RUOUG Seminar

SQLgt oradebug peek 200222A0 24

[200222A0 200222B8) = 00000016 00000001 000001D0 00000007

pid^ gets latch level

SQLgt oradebug peek 0xampladdr 24

[2000C814 2000C82C) = 2000001F 0000007C 0000007E 00000006 pid^ gets latch level

983139983151983150983156 983155983153983148983152983148983157983155 983150983151983148983151983143 9831409831419831499831519831350128

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2982

LATCH FIXED TABLES

bull Before Oracle 11g all the latches were externalized to SQL inv$latch (x$ksllt)

bull In 11g parent (fixed SGA) latches are externalized inv$latch_parent (x$kslltr_parent)

bull In 11g child (allocated on startup) latches are externalized in

v$latch_children (x$kslltr_children) bull Oracle 11g v$latch (x$kslltr) merges these data Beware select

from x$kslltr acquires all parent latches and their children

bull Only part of latch structure is visible in x$ksllthellip fixed tables ndash kslltwhr kslltwhy ndash ldquowhererdquo and ldquowhyrdquo latch was acquired last time

ndash kslltwgt hellip - statistics

ndash kslltlvl hellip - attributes

ndash kslltwtt ndash latch wait time

RDTEX | Latch internals | RUOUG Seminar

29

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3082

CUMULATIVE V$LATCH STATISTIC COUNTERS

RDTEX | Latch internals | RUOUG Seminar

Statistics x$ksllt When and how it increments

GETS kslltwgt ldquo++rdquo after wait mode latch get

MISSES kslltwff ldquo++rdquo after wait get if it was missed

SLEEPS kslltwsl ldquo+number_of_sleepsrdquo during get

SPIN_GETS ksllthst0 ldquo++rdquo if get was missed but not slept

WAIT_TIME kslltwtt ldquo+wait_timerdquo after latch get

IMMEDIATE_GETS kslltngt ldquo++rdquo after nowait mode latch get Is notincremented on miss

IMMEDIATE_MISSES kslltnfa ldquo++rdquo if nowait mode get was missed

30

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3182

MANY STATISTICS WERE LOST IN 102

In 102 Oracle made obsolete many useful latch statistics

Latch size was decreased and the following fields always contain 0now in v$latch

ndash v$latchwaiters_woken

ndash v$latchwaits_holding_latch

ndash v$latchsleep1hellipsleep10

ndash and many undocumented statistics

This was mentioned in 102 Reference

RDTEX | Latch internals | RUOUG Seminar

31

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3282

X$KSLLD - LATCH DESCRIPTORS

bull Flags and attributes describing each latch are stored in kslldt arrayof kslld structures

bull [K]ernel [S]ervice [L]atch [L]ock [D]escriptor

bull Some but not all of kslld fields are externalized via x$kslld andv$latchname

bull Located in the PGA

RDTEX | Latch internals | RUOUG Seminar

32

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382

LATCH ATTRIBUTES

bull Each latch have at least the following attributes in kslldt

ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared

ndash PAR Is the latch Solitary or Parent for the family of child latches

ndash G2C Can two child latches be simultaneously requested in wait mode

using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92

ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above

ndash Level 0-14 Used to prevent latch deadlocks

ndash Class 0-7 Spin and wait class assigned to the latch 92 and above

RDTEX | Latch internals | RUOUG Seminar

33

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482

LATCH ATTRIBUTES BY ORACLE VERSION

RDTEX | Latch internals | RUOUG Seminar

11203 553 154 72 - 6 93

Oracle Number of latches PAR G2C LNG UFS SHARED

7340 53 14 2 3 - -

8063 80 21 7 3 - 38174 152 48 19 4 - 9

9208 242 79 37 - - 19

10202 385 114 55 - 4 47

10203 388 117 58 - 4 48

10204 394 117 59 - 4 50

11106 496 145 67 - 6 81

11107 502 145 67 - 6 83

11201 535 149 70 - 6 86

11202 551 153 72 - 6 91

34

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582

LATCH LEVELS

bull Every latch has a level in the range 0-14 to avoid deadlocks

bull Process can request a latch X in wait mode after obtaining latch Y if and

only if level X gt level Y

bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y

bull If these rules are broken the Oracle process raises

ORA-600[504] - trying to obtain another latch at incompatible level Note 281041

ORA-600[525] - trying to obtain another latch at the same level Note 1388881

ORA-600[526] - latch child order violation Note 1388891

RDTEX | Latch internals | RUOUG Seminar

35

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682

PARENT AND CHILD LATCHES

bull Latch can be a parent child or solitary latch

bull Both parent and child latches share the same latch name

bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss

bull Child latches are allocated during instance startup

bull The parent latch can be gotten independently

bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s

RDTEX | Latch internals | RUOUG Seminar

(latch info) wait_event=0 bits=10

holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl

Context saved from call 0

state=busy

36

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782

LATCH TREES

ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches

Direct SGA access program output for 9206 instance with too smallshared pool

RDTEX | Latch internals | RUOUG Seminar

ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1

waiter ospid 129 sid 72 pid 45

holding a154b7120 library cache (157) level=5 child=17 kglupc child

waiter ospid 18255 sid 65 pid 930

waiter ospid 6690 sid 554 pid 1654

waiter ospid 4685 sid 879 pid 1034

hellip

waiter ospid 29749 sid 180 pid 155

holding a154b7db8 library cache (157) level=5 child=4 kglupc child

waiter ospid 13104 sid 281 pid 220

waiter ospid 24089 sid 565 pid 636

waiter ospid 25002 sid 621 pid 1481

waiter ospid 16930 sid 1046 pid 783hellip

37

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882

LATCH CLASSES

bull Classes with different spin and wait policies

bull Up to 8 classes By default all the latches belong to class 0

bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2

bull Latch can be assigned to class by its number For example

ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000

ndash _LATCH_CLASSES=46

bull Will be discussed later

RDTEX | Latch internals | RUOUG Seminar

SQLgt select from x$ksllclass

INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4

---- ------ ----- -------- ------ ------ ------ ------ ------

0 20000 0 1 8000 8000 8000 8000 8000

hellip

6 100 12 1000 2000 3000 4000 4000 4000

38

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982

LATCH ACQUISITION IN NO-WAIT MODE

bull kslgetl(laddr0whywhere) ndash Exclusive latch get

bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep

ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches

ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches

bull IMMEDIATE_ statistics are updated

bull Used when many candidate latches are available to bypass the

latch level rule or as part of CBC chain examination

RDTEX | Latch internals | RUOUG Seminar

sqlplus nolog demo_03sql39

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082

LATCH WAITS

RDTEX | Latch internals | RUOUG Seminar 40

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182

WAITING FOR THE LATCH

RDTEX | Latch internals | RUOUG Seminar

983123 G A

C983120983125 1 C983120983125 2

983120983154983151983139983141983155983155 A

983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)

983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144

983116983137983156983139983144

983120983154983151983139983141983155983155 B

41

RDTEX | L h i l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282

LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION

bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries

again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of

times

bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep

bull A sleep corresponds to a wait for the latch free wait event

bull Initially a process sleeps for 001 s This time is doubled in every

subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip

RDTEX | Latch internals | RUOUG Seminar

Still did not found any versionwhich used this

May be it was lt= Oracle 6

42

RDTEX | L t h i t l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382

LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81

Latch gets with wait (kslgetl(laddress1hellip))

ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times

ndash Sleep on latch free event with exponential backoff

ndash Repeat

RDTEX | Latch internals | RUOUG Seminar

43

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482

8I LATCH SPIN CODE FLOW USING DTRACE

RDTEX | Latch internals | RUOUG Seminar

bull hellip Event 10046 trace

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2

983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29

983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270

sqlplus nolog demo_04_8sql 29

44

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582

POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF

bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec

RDTEX | Latch internals | RUOUG Seminar

12 ]2 )1[(

minus= +wait N

timeout

45

bull Most waits were for nothing ndash latch already was free

bull Latch utilization can not be high

bull Unnecessary spins provoke CPU thrashing

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682

92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE

sqlplus nolog demo_05sql 45

RDTEX | Latch internals | RUOUG Seminar

kslgetl(0x50006318 1)

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt kslges(0x50006318 ) -wait latch get

-gt skgslsgts(0x50006318 ) -spin latch get

-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT

-gt kskthbwt(0x0)

-gt kslwlmod() - set up Wait List

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt skgpwwait -sleep latch getsemop(11 17-10 1)

983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140

46

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782

RELIABLE LATCH WAITS

bull Hidden latch wait revolution that we missed Latch relies on waitposting

bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout

bull Exclusive latch spins 20000 cycles by default

bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock

bull Latches assigned to non-default class wait until timeout

bull Version 90 was transient and used mix of backoff and wait posting

This is why _LATCH_WAIT_POSTING parameter disappeared in9i

bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin

RDTEX | Latch internals | RUOUG Seminar

47

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882

IMAGINE THE TIME MICROSCOPE

| a a s | a

Reality One million zoomed time

1 us 1 sec

1 ms 17 min1 sec 115 days

Light speed (300000 kmsec) Sonic speed (300 msec)

CPU tick (2 GHz) 00005 sec

Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec

Max spin time for shared latch (~220005 ticks) ~10 sec

Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec

Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec

OS context switch time (10us-1ms) 10sec-17min

48

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982

LATCH WAITS UNDER TIME MICROSCOPE

Old 8i latch algorithmSpin for latch during 5 seconds

Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min

hellipSpin again for 5 seconds

Sleep again for 23 days (2 sec) hellip

New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)

If spin not succeeded go to sleep until post

According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)

| |

49

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082

_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES

bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and

_LATCH_CLASS_0 parameters

bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000

bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter

bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch

50

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182

TIMED LATCH WAIT POSTING

bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel

bull This can lead to instance hang ndash process will never be woken up

bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout

bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem

bull Generic AIX ports always use timed latch wait posting

bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux

51

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282

FINE GRAIN LATCH TUNING

bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting

bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7

bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo

microseconds using pollsys() (not semtimedop()) system call

ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times

Loop up to ldquoSpinsrdquo cycles

Yield CPU using yield() (or sched_yield())

ndash Sleep for ldquoSleepXrdquo usecs

ndash Then spin again hellip

52

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382

TRACES OF NONDEFAULT CLASS LATCH

983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152

991270

9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000

991270 E983158983141983150983156 10046 983156983154983137983139983141

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482

PERFORMANCE OF LATCH CLASSES

Row cache objects latch contention testcase on ODA (24 SMT)54

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582

IS THE _SPIN_COUNT REALLY STATIC

bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch

waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used

ldquocache buffers chainrdquo latch contention

bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass

(_LATCH_CLASS_X) By default ndash 20000

bull Shared latch spin can be tuned by _SPIN_COUNT value By

default ndash 2000 However this is not trivial

55

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682

SHARED LATCH BEHAVE LIKE ENQUEUE

bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80

bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests

bull This blocking achieved by special 0x40000000 bit in the latch

value This bit is a flag that some incompatible latch acquisitionis in progress

bull Queued processes will be woken up one by one on latch release

bull X mode latch gets effectively serialize the shared latch

bull See my blog for description of experiments

56

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782

TRACE OF SHARED LATCH WAIT

Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the

latch wait list and then spins again

983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141

983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156

983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983147983155983148983159983148983149983151983140()

983147983155983148983155983147983143983155(098316050009BAC )

983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140

983155983141983149983151983152(27)

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882

SHARED LATCH ACQUISITION

bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned

bull X mode shared latch get spins by default up to 4000 cycles

bull S mode does not spin at all (or spins in unknown way)

S mode get X mode get

Held in S mode Compatible 2_SPIN_COUNT

Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT

58

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982

LATCH RELEASE

bull Free the latch ndash kslfre(laddr)

bull Oracle process releases the latch nonatomically

bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process

bull This requires less bus invalidation and ensures propagation of latch

release to other local cachesbull Not fair policy Spinners on the local CPU board have the

preference

bull Then process posts the first process in the list of waiters

59

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082

LATCH CONTENTION

RDTEX | Latch internals | RUOUG Seminar60

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182

LATCH CONTENTION

bull Occurs when a latch is required by several processes at thesame time

bull Mostly due to bugs or abnormal application behavior

bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp

bull To diagnose the latch contention we need to investigate thelatch statistics [8]

61

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282

DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS

Latch requests arrival rate

Immediate gets efficiency

Latch sleeps ratio

Latch wait time per second

Latch spin efficiencymisses

getsspin∆

∆= _σ

misses

sleepsK

∆=

time

gets

∆=λ

gets

misses

∆= ρ

time

timewait W

∆=

_

Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics

62

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382

SAMPLED AVERAGES

By sampling of x$ksupr we can estimate

ndash Length of latch wait queue

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS

ndash Number of spinning processes in 10g

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS

Sampling of v$lathholder estimates latch utilization

Using DTrace we can measure

ndash Latch acquisition and holding times

ndash Actual spin count

w L

s N

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482

DERIVED LATCH STATISTICS

Latch utilization (PASTA)

Average holding time

Length of latch wait list

Recurrent sleeps ratio

Latch acquisition time

time

timeholdinglatchU

∆=asymp

__ ρ

Re_100

___

questsGet

TimeSnap MissGet Pct S

lowast==

λ

W L =

κ

κ σ 1minus+

)(1

W N T saq += minusλ

64

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 12: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1282

THE TOOLS

RDTEX | Latch internals | RUOUG Seminar12

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1382

DTRACE SOLARIS DYNAMIC TRACING FRAMEWORK

DTrace allows us to investigate how Oracle latches perform in real time ndash Create triggers on any event inside Solaris and function call inside Oracle

providermodulefunctionname

pid1910oraclekslgetlentry ndash Write trigger bodies ndash actions May read and change any address location ndash Can count the latch spins trace the latch waits perform experiments ndash Measure times and distributions up to microseconds precision

RDTEX | Latch internals | RUOUG Seminar

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313500

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1482

ORADEBUG INTERNAL ORACLE DEBUGGER

RDTEX | Latch internals | RUOUG Seminar

bull oradebug call Allows us invoke any internal Oracle function manually

SQLgt oradebug peek 200222A0 24

[200222A0 200222B8) = 00000016 00000001 000001D0 00000007

SQLgt oradebug call kslgetl 0x5001A020 1 2 3Function returned 1

bull oradebug peek Examines contents of any memory location

bull oradebug poke Modifies memory

bull oradebug watch Watch a region of memory

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1582

LATCH CONTENTION TESTCASES

RDTEX | Latch internals | RUOUG Seminar

15

create table latch_contention as select from dualhellipfor i in 11000000loopselect dummy into a from latch_contention

end loop

bull Exclusive and shared latches behave differentlybull Contention for shared cache buffer chains latch arises when manysessions scan the same block concurrently and _fastpin_enable=0

bull To induce the contention for exclusive ldquorow cache objectsrdquo latch protectingdc_users row cache just add several dynamic RLS policies

exec dbms_rlsadd_policy(systemlatch_contentionrls_1 hellip

exec dbms_rlsadd_policy(systemlatch_contentionrls_2 helliphellip

bull These dummy policies generate predicates 1=1bull Many other contention scenarios possible

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313509831351

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1682

ROW CACHE LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

16

Elapsed and CPU times vs number of threadsOracle Database Appliance (24 SMT)

Latch wait

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1782

HOW ORACLEREQUESTS THE LATCH

RDTEX | Latch internals | RUOUG Seminar17

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1882

DTRACE REVEALS LATCH INTERFACE ROUTINES

Oracle calls the following functions to acquire the latch

ndash kslgetl(laddr wait why where) - get exclusive latch ndash kslg2c(l1l2trcwhy where) - get two excl child latches

ndash kslgetsl (laddrwaitwhywheremode) - get shared latch

bull 11g - ksl_get_shared_latch (hellip)

ndash kslg2cs(l1l2modetrcwhy where)) - get two shr child latches

ndash kslgpl(laddrcommentwhywhere) - get parent and all childs

ndash kslfre(laddr) - free the latch

Oracle give us possibility to do the same by oradebug call

RDTEX | Latch internals | RUOUG Seminar

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313501

983125983155983141 32983085983138983145983156 983152983151983154983156983155

64983138983145983156 O983154983137983139983148983141 983139983151983154983154983157983152983156983155 983156983144983141 983137983154983143983157983149983141983150983156983155 983151983142 983151983154983137983140983141983138983157983143 983139983137983148983148 983145983150 10204 983137983150983140 983137983138983151983158983141

18

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1982

ldquoWHERErdquo AND ldquoWHYrdquo ORACLE GETS THE LATCH

To request the latch Oracle kernel routines need

bull laddres ndash address of latch in SGAbull wait ndash flag for no-wait (0) or wait (1) latch acquisition

bull where ndash integer code for location from where the latch is acquired

Oracle lists possible latch ldquowhererdquo values in x$ksllwindxbull why - context of why the latch is acquiring at this ldquowhererdquo It

contains DBA for CBC latches (Tanel Poder) SGA chunk addressfor shared pool latch session address etchellip ldquoWhyrdquo meaning for

some ldquowhererdquo can be found in x$ksllwksllwlbl

bull mode ndash requesting state for shared lathes

ndash 8 SHARED mode 16 EXCLUSIVE mode

RDTEX | Latch internals | RUOUG Seminar

19

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2082

LIST OF LATCH ldquoWHERErdquo LOCATIONS

X$KSLLW ndash List of latch [W]here Part of v$latch_misses

ndash ksllwnam ndash Kernel function name(ldquokghupr1rdquo ldquoksqdelrdquo etc) ndash ksllwlbl ndash Description of ldquowhyrdquo context

Join this code on indx to

ndash x$ksuprlatksulawhr (v$latchholder)ndash ldquowhererdquo value that wasused by latch holder

ndash x$kslltkslltwhr (v$latch) ndash ldquowhererdquo that was used by lastsuccessful latch acquisition

ndash x$ksuprksllawer (v$process) ndash ldquowhererdquo that was used byprocess is being waiting for latch

ndash x$kslwscindx (v$latch_misses) ndash latch sleep statistics by

ldquowhererdquo

RDTEX | Latch internals | RUOUG Seminar

20

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2182

V$LATCH_MISSES

bull Based on X$KSLWSC - No[W]ait and [S]leep [C]ount stats byContext Documentation is rather obscure

bull The following counters are incremented on each latch sleep

sleep_count (ldquowhererdquo the latch was held)

wtr_slp_count (ldquowhererdquo the latch get was requested)

bull nwfail_count is incremented only for session which SID is equalto _LATCH_MISS_STAT_SID parameter

bull longhold_count is incremented for shared latches only when

someone held the latch at this ldquowhererdquo for the entire duration ofsleep

RDTEX | Latch internals | RUOUG Seminar

21

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2282

LATCH IS HOLDING BY PROCESS NOT SESSION

bull

RDTEX | Latch internals | RUOUG Seminar

983123983156983154983157983139983156 983147983155983157983152983154 983163

991270

983123983156983154983157983139983156 983147983155983148983148983137983163983147983155983148983148983156 98314798315598314898314898313798314898313798315698313114983133

983165

991270983165

983120983154983151983139983141983155983155 983142983145983160983141983140 983137983154983154983137983161

983158$983152983154983151983139983141983155983155 983085gt 983160$983147983155983157983152983154

983123983156983154983157983139983156 983147983155983148983148983156983163

991270

983165

L983145983155983156 983151983142 983137983148983148 983148983137983156983139983144983141983155

983158$983148983137983156983139983144 983085gt983160$983147983155983148983148983156

Each process has an array of references to the latches it is holding

22

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2382

KSLLA IS HIDDEN IN V$PROCESS

bull The process latching info is the kslla structure embedded in theprocess state object

bull Some kslla fields were externalized to SQL

bull kslla contains array of latches holding by process at each level andinformation about the state of any current attempt to acquire a

latch Not externalized directly

bull When you query v$latchholder x$ksuprlat Oracle scansv$processx$ksupr in order to find all the processes that are

holding the latchesbull Tanel Poder uses high frequency sampling of v$latchholder to

systematically troubleshoot latch contention

RDTEX | Latch internals | RUOUG Seminar

23

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2482

X$KSUPRKSLLA FIELDS INSTRUMENT THE LATCH GET

bull ksllalaq ndash address of latch acquiring Populated during immediateget (and spin before 11g)

bull ksllawat - latch being waited for This is v$processlatchwait

bull ksllawhy ndash ldquowhyrdquo for the latch being waited for

bull ksllawere ndash ldquowhererdquo for the latch being waited for

bull ksllalow ndash bit array of levels of currently holding latches

bull ksllaspn - latch this process is spinning on v$processlatchspinNot populated since 81

bull ksllaps - inter-process post statistics

RDTEX | Latch internals | RUOUG Seminar

24

Generic ldquolatch freerdquo and 27 latch specific ldquolatch free rdquo wait events areassociated with sleep only Spin is invisible to Oracle Wait Interface

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2582

THE LATCH STRUCTURE ndash KSLLT

RDTEX | Latch internals | RUOUG Seminar

struct ksllt

ltLatchgt

ldquowhererdquo and ldquowhyrdquo

Level latch class other attributes

Statistics

Latch wait list header

hellip

25

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2682

LATCH SIZE BY VERSION

nix 32bit nix 64bit Windows 32bit

734 92 - 120

806 104 - 104817 104 144 104

901 200 160

920 196 240 200

1010 256 208

1020 - 11203 100 160 104

RDTEX | Latch internals | RUOUG Seminar

x$ksmfsv ndash list of all fixed SGA variables983123E983116EC983124 DI983123983124I983118C983124 983147983155983149983142983155983155983145983162

F983122983119983117 983160$983147983155983149983142983155983158

983127HE983122E 983147983155983149983142983155983156983161983152 = 983147983155983148983148983156

Latch structure was bigger in 101 due to additional latch statistics

26

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2782

ORACLE LATCH IS NOT JUST A SINGLE MEMORY LOCATION

bull Before 11g Value of first latch byte (word for shared latches) wasused to determine latch state

0x00 ndash latch is free

0xFF ndash exclusive latch is busy Was 0x01 in Oracle 7

0x010x02hellip - shared latch holding by 12 hellip processes simultaneously

0x20000000 | pid - shared latch holding exclusively

bull In 11g first latch word shows the pid of the latch holder

0x00 ndash latch is free0x12 ndash Oracle process with pid 18 holds the exclusive latch

RDTEX | Latch internals | RUOUG Seminar

27

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2882

BUSY ORACLE 11G LATCH IN MEMORY

bull 112 Exclusive latch (32-bit platform)

bull 112 Shared latch EXCLUSIVE mode

RDTEX | Latch internals | RUOUG Seminar

SQLgt oradebug peek 200222A0 24

[200222A0 200222B8) = 00000016 00000001 000001D0 00000007

pid^ gets latch level

SQLgt oradebug peek 0xampladdr 24

[2000C814 2000C82C) = 2000001F 0000007C 0000007E 00000006 pid^ gets latch level

983139983151983150983156 983155983153983148983152983148983157983155 983150983151983148983151983143 9831409831419831499831519831350128

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2982

LATCH FIXED TABLES

bull Before Oracle 11g all the latches were externalized to SQL inv$latch (x$ksllt)

bull In 11g parent (fixed SGA) latches are externalized inv$latch_parent (x$kslltr_parent)

bull In 11g child (allocated on startup) latches are externalized in

v$latch_children (x$kslltr_children) bull Oracle 11g v$latch (x$kslltr) merges these data Beware select

from x$kslltr acquires all parent latches and their children

bull Only part of latch structure is visible in x$ksllthellip fixed tables ndash kslltwhr kslltwhy ndash ldquowhererdquo and ldquowhyrdquo latch was acquired last time

ndash kslltwgt hellip - statistics

ndash kslltlvl hellip - attributes

ndash kslltwtt ndash latch wait time

RDTEX | Latch internals | RUOUG Seminar

29

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3082

CUMULATIVE V$LATCH STATISTIC COUNTERS

RDTEX | Latch internals | RUOUG Seminar

Statistics x$ksllt When and how it increments

GETS kslltwgt ldquo++rdquo after wait mode latch get

MISSES kslltwff ldquo++rdquo after wait get if it was missed

SLEEPS kslltwsl ldquo+number_of_sleepsrdquo during get

SPIN_GETS ksllthst0 ldquo++rdquo if get was missed but not slept

WAIT_TIME kslltwtt ldquo+wait_timerdquo after latch get

IMMEDIATE_GETS kslltngt ldquo++rdquo after nowait mode latch get Is notincremented on miss

IMMEDIATE_MISSES kslltnfa ldquo++rdquo if nowait mode get was missed

30

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3182

MANY STATISTICS WERE LOST IN 102

In 102 Oracle made obsolete many useful latch statistics

Latch size was decreased and the following fields always contain 0now in v$latch

ndash v$latchwaiters_woken

ndash v$latchwaits_holding_latch

ndash v$latchsleep1hellipsleep10

ndash and many undocumented statistics

This was mentioned in 102 Reference

RDTEX | Latch internals | RUOUG Seminar

31

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3282

X$KSLLD - LATCH DESCRIPTORS

bull Flags and attributes describing each latch are stored in kslldt arrayof kslld structures

bull [K]ernel [S]ervice [L]atch [L]ock [D]escriptor

bull Some but not all of kslld fields are externalized via x$kslld andv$latchname

bull Located in the PGA

RDTEX | Latch internals | RUOUG Seminar

32

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382

LATCH ATTRIBUTES

bull Each latch have at least the following attributes in kslldt

ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared

ndash PAR Is the latch Solitary or Parent for the family of child latches

ndash G2C Can two child latches be simultaneously requested in wait mode

using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92

ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above

ndash Level 0-14 Used to prevent latch deadlocks

ndash Class 0-7 Spin and wait class assigned to the latch 92 and above

RDTEX | Latch internals | RUOUG Seminar

33

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482

LATCH ATTRIBUTES BY ORACLE VERSION

RDTEX | Latch internals | RUOUG Seminar

11203 553 154 72 - 6 93

Oracle Number of latches PAR G2C LNG UFS SHARED

7340 53 14 2 3 - -

8063 80 21 7 3 - 38174 152 48 19 4 - 9

9208 242 79 37 - - 19

10202 385 114 55 - 4 47

10203 388 117 58 - 4 48

10204 394 117 59 - 4 50

11106 496 145 67 - 6 81

11107 502 145 67 - 6 83

11201 535 149 70 - 6 86

11202 551 153 72 - 6 91

34

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582

LATCH LEVELS

bull Every latch has a level in the range 0-14 to avoid deadlocks

bull Process can request a latch X in wait mode after obtaining latch Y if and

only if level X gt level Y

bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y

bull If these rules are broken the Oracle process raises

ORA-600[504] - trying to obtain another latch at incompatible level Note 281041

ORA-600[525] - trying to obtain another latch at the same level Note 1388881

ORA-600[526] - latch child order violation Note 1388891

RDTEX | Latch internals | RUOUG Seminar

35

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682

PARENT AND CHILD LATCHES

bull Latch can be a parent child or solitary latch

bull Both parent and child latches share the same latch name

bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss

bull Child latches are allocated during instance startup

bull The parent latch can be gotten independently

bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s

RDTEX | Latch internals | RUOUG Seminar

(latch info) wait_event=0 bits=10

holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl

Context saved from call 0

state=busy

36

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782

LATCH TREES

ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches

Direct SGA access program output for 9206 instance with too smallshared pool

RDTEX | Latch internals | RUOUG Seminar

ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1

waiter ospid 129 sid 72 pid 45

holding a154b7120 library cache (157) level=5 child=17 kglupc child

waiter ospid 18255 sid 65 pid 930

waiter ospid 6690 sid 554 pid 1654

waiter ospid 4685 sid 879 pid 1034

hellip

waiter ospid 29749 sid 180 pid 155

holding a154b7db8 library cache (157) level=5 child=4 kglupc child

waiter ospid 13104 sid 281 pid 220

waiter ospid 24089 sid 565 pid 636

waiter ospid 25002 sid 621 pid 1481

waiter ospid 16930 sid 1046 pid 783hellip

37

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882

LATCH CLASSES

bull Classes with different spin and wait policies

bull Up to 8 classes By default all the latches belong to class 0

bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2

bull Latch can be assigned to class by its number For example

ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000

ndash _LATCH_CLASSES=46

bull Will be discussed later

RDTEX | Latch internals | RUOUG Seminar

SQLgt select from x$ksllclass

INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4

---- ------ ----- -------- ------ ------ ------ ------ ------

0 20000 0 1 8000 8000 8000 8000 8000

hellip

6 100 12 1000 2000 3000 4000 4000 4000

38

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982

LATCH ACQUISITION IN NO-WAIT MODE

bull kslgetl(laddr0whywhere) ndash Exclusive latch get

bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep

ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches

ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches

bull IMMEDIATE_ statistics are updated

bull Used when many candidate latches are available to bypass the

latch level rule or as part of CBC chain examination

RDTEX | Latch internals | RUOUG Seminar

sqlplus nolog demo_03sql39

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082

LATCH WAITS

RDTEX | Latch internals | RUOUG Seminar 40

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182

WAITING FOR THE LATCH

RDTEX | Latch internals | RUOUG Seminar

983123 G A

C983120983125 1 C983120983125 2

983120983154983151983139983141983155983155 A

983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)

983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144

983116983137983156983139983144

983120983154983151983139983141983155983155 B

41

RDTEX | L h i l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282

LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION

bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries

again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of

times

bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep

bull A sleep corresponds to a wait for the latch free wait event

bull Initially a process sleeps for 001 s This time is doubled in every

subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip

RDTEX | Latch internals | RUOUG Seminar

Still did not found any versionwhich used this

May be it was lt= Oracle 6

42

RDTEX | L t h i t l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382

LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81

Latch gets with wait (kslgetl(laddress1hellip))

ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times

ndash Sleep on latch free event with exponential backoff

ndash Repeat

RDTEX | Latch internals | RUOUG Seminar

43

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482

8I LATCH SPIN CODE FLOW USING DTRACE

RDTEX | Latch internals | RUOUG Seminar

bull hellip Event 10046 trace

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2

983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29

983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270

sqlplus nolog demo_04_8sql 29

44

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582

POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF

bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec

RDTEX | Latch internals | RUOUG Seminar

12 ]2 )1[(

minus= +wait N

timeout

45

bull Most waits were for nothing ndash latch already was free

bull Latch utilization can not be high

bull Unnecessary spins provoke CPU thrashing

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682

92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE

sqlplus nolog demo_05sql 45

RDTEX | Latch internals | RUOUG Seminar

kslgetl(0x50006318 1)

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt kslges(0x50006318 ) -wait latch get

-gt skgslsgts(0x50006318 ) -spin latch get

-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT

-gt kskthbwt(0x0)

-gt kslwlmod() - set up Wait List

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt skgpwwait -sleep latch getsemop(11 17-10 1)

983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140

46

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782

RELIABLE LATCH WAITS

bull Hidden latch wait revolution that we missed Latch relies on waitposting

bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout

bull Exclusive latch spins 20000 cycles by default

bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock

bull Latches assigned to non-default class wait until timeout

bull Version 90 was transient and used mix of backoff and wait posting

This is why _LATCH_WAIT_POSTING parameter disappeared in9i

bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin

RDTEX | Latch internals | RUOUG Seminar

47

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882

IMAGINE THE TIME MICROSCOPE

| a a s | a

Reality One million zoomed time

1 us 1 sec

1 ms 17 min1 sec 115 days

Light speed (300000 kmsec) Sonic speed (300 msec)

CPU tick (2 GHz) 00005 sec

Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec

Max spin time for shared latch (~220005 ticks) ~10 sec

Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec

Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec

OS context switch time (10us-1ms) 10sec-17min

48

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982

LATCH WAITS UNDER TIME MICROSCOPE

Old 8i latch algorithmSpin for latch during 5 seconds

Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min

hellipSpin again for 5 seconds

Sleep again for 23 days (2 sec) hellip

New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)

If spin not succeeded go to sleep until post

According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)

| |

49

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082

_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES

bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and

_LATCH_CLASS_0 parameters

bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000

bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter

bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch

50

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182

TIMED LATCH WAIT POSTING

bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel

bull This can lead to instance hang ndash process will never be woken up

bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout

bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem

bull Generic AIX ports always use timed latch wait posting

bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux

51

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282

FINE GRAIN LATCH TUNING

bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting

bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7

bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo

microseconds using pollsys() (not semtimedop()) system call

ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times

Loop up to ldquoSpinsrdquo cycles

Yield CPU using yield() (or sched_yield())

ndash Sleep for ldquoSleepXrdquo usecs

ndash Then spin again hellip

52

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382

TRACES OF NONDEFAULT CLASS LATCH

983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152

991270

9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000

991270 E983158983141983150983156 10046 983156983154983137983139983141

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482

PERFORMANCE OF LATCH CLASSES

Row cache objects latch contention testcase on ODA (24 SMT)54

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582

IS THE _SPIN_COUNT REALLY STATIC

bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch

waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used

ldquocache buffers chainrdquo latch contention

bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass

(_LATCH_CLASS_X) By default ndash 20000

bull Shared latch spin can be tuned by _SPIN_COUNT value By

default ndash 2000 However this is not trivial

55

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682

SHARED LATCH BEHAVE LIKE ENQUEUE

bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80

bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests

bull This blocking achieved by special 0x40000000 bit in the latch

value This bit is a flag that some incompatible latch acquisitionis in progress

bull Queued processes will be woken up one by one on latch release

bull X mode latch gets effectively serialize the shared latch

bull See my blog for description of experiments

56

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782

TRACE OF SHARED LATCH WAIT

Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the

latch wait list and then spins again

983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141

983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156

983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983147983155983148983159983148983149983151983140()

983147983155983148983155983147983143983155(098316050009BAC )

983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140

983155983141983149983151983152(27)

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882

SHARED LATCH ACQUISITION

bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned

bull X mode shared latch get spins by default up to 4000 cycles

bull S mode does not spin at all (or spins in unknown way)

S mode get X mode get

Held in S mode Compatible 2_SPIN_COUNT

Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT

58

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982

LATCH RELEASE

bull Free the latch ndash kslfre(laddr)

bull Oracle process releases the latch nonatomically

bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process

bull This requires less bus invalidation and ensures propagation of latch

release to other local cachesbull Not fair policy Spinners on the local CPU board have the

preference

bull Then process posts the first process in the list of waiters

59

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082

LATCH CONTENTION

RDTEX | Latch internals | RUOUG Seminar60

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182

LATCH CONTENTION

bull Occurs when a latch is required by several processes at thesame time

bull Mostly due to bugs or abnormal application behavior

bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp

bull To diagnose the latch contention we need to investigate thelatch statistics [8]

61

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282

DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS

Latch requests arrival rate

Immediate gets efficiency

Latch sleeps ratio

Latch wait time per second

Latch spin efficiencymisses

getsspin∆

∆= _σ

misses

sleepsK

∆=

time

gets

∆=λ

gets

misses

∆= ρ

time

timewait W

∆=

_

Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics

62

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382

SAMPLED AVERAGES

By sampling of x$ksupr we can estimate

ndash Length of latch wait queue

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS

ndash Number of spinning processes in 10g

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS

Sampling of v$lathholder estimates latch utilization

Using DTrace we can measure

ndash Latch acquisition and holding times

ndash Actual spin count

w L

s N

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482

DERIVED LATCH STATISTICS

Latch utilization (PASTA)

Average holding time

Length of latch wait list

Recurrent sleeps ratio

Latch acquisition time

time

timeholdinglatchU

∆=asymp

__ ρ

Re_100

___

questsGet

TimeSnap MissGet Pct S

lowast==

λ

W L =

κ

κ σ 1minus+

)(1

W N T saq += minusλ

64

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 13: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1382

DTRACE SOLARIS DYNAMIC TRACING FRAMEWORK

DTrace allows us to investigate how Oracle latches perform in real time ndash Create triggers on any event inside Solaris and function call inside Oracle

providermodulefunctionname

pid1910oraclekslgetlentry ndash Write trigger bodies ndash actions May read and change any address location ndash Can count the latch spins trace the latch waits perform experiments ndash Measure times and distributions up to microseconds precision

RDTEX | Latch internals | RUOUG Seminar

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313500

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1482

ORADEBUG INTERNAL ORACLE DEBUGGER

RDTEX | Latch internals | RUOUG Seminar

bull oradebug call Allows us invoke any internal Oracle function manually

SQLgt oradebug peek 200222A0 24

[200222A0 200222B8) = 00000016 00000001 000001D0 00000007

SQLgt oradebug call kslgetl 0x5001A020 1 2 3Function returned 1

bull oradebug peek Examines contents of any memory location

bull oradebug poke Modifies memory

bull oradebug watch Watch a region of memory

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1582

LATCH CONTENTION TESTCASES

RDTEX | Latch internals | RUOUG Seminar

15

create table latch_contention as select from dualhellipfor i in 11000000loopselect dummy into a from latch_contention

end loop

bull Exclusive and shared latches behave differentlybull Contention for shared cache buffer chains latch arises when manysessions scan the same block concurrently and _fastpin_enable=0

bull To induce the contention for exclusive ldquorow cache objectsrdquo latch protectingdc_users row cache just add several dynamic RLS policies

exec dbms_rlsadd_policy(systemlatch_contentionrls_1 hellip

exec dbms_rlsadd_policy(systemlatch_contentionrls_2 helliphellip

bull These dummy policies generate predicates 1=1bull Many other contention scenarios possible

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313509831351

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1682

ROW CACHE LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

16

Elapsed and CPU times vs number of threadsOracle Database Appliance (24 SMT)

Latch wait

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1782

HOW ORACLEREQUESTS THE LATCH

RDTEX | Latch internals | RUOUG Seminar17

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1882

DTRACE REVEALS LATCH INTERFACE ROUTINES

Oracle calls the following functions to acquire the latch

ndash kslgetl(laddr wait why where) - get exclusive latch ndash kslg2c(l1l2trcwhy where) - get two excl child latches

ndash kslgetsl (laddrwaitwhywheremode) - get shared latch

bull 11g - ksl_get_shared_latch (hellip)

ndash kslg2cs(l1l2modetrcwhy where)) - get two shr child latches

ndash kslgpl(laddrcommentwhywhere) - get parent and all childs

ndash kslfre(laddr) - free the latch

Oracle give us possibility to do the same by oradebug call

RDTEX | Latch internals | RUOUG Seminar

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313501

983125983155983141 32983085983138983145983156 983152983151983154983156983155

64983138983145983156 O983154983137983139983148983141 983139983151983154983154983157983152983156983155 983156983144983141 983137983154983143983157983149983141983150983156983155 983151983142 983151983154983137983140983141983138983157983143 983139983137983148983148 983145983150 10204 983137983150983140 983137983138983151983158983141

18

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1982

ldquoWHERErdquo AND ldquoWHYrdquo ORACLE GETS THE LATCH

To request the latch Oracle kernel routines need

bull laddres ndash address of latch in SGAbull wait ndash flag for no-wait (0) or wait (1) latch acquisition

bull where ndash integer code for location from where the latch is acquired

Oracle lists possible latch ldquowhererdquo values in x$ksllwindxbull why - context of why the latch is acquiring at this ldquowhererdquo It

contains DBA for CBC latches (Tanel Poder) SGA chunk addressfor shared pool latch session address etchellip ldquoWhyrdquo meaning for

some ldquowhererdquo can be found in x$ksllwksllwlbl

bull mode ndash requesting state for shared lathes

ndash 8 SHARED mode 16 EXCLUSIVE mode

RDTEX | Latch internals | RUOUG Seminar

19

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2082

LIST OF LATCH ldquoWHERErdquo LOCATIONS

X$KSLLW ndash List of latch [W]here Part of v$latch_misses

ndash ksllwnam ndash Kernel function name(ldquokghupr1rdquo ldquoksqdelrdquo etc) ndash ksllwlbl ndash Description of ldquowhyrdquo context

Join this code on indx to

ndash x$ksuprlatksulawhr (v$latchholder)ndash ldquowhererdquo value that wasused by latch holder

ndash x$kslltkslltwhr (v$latch) ndash ldquowhererdquo that was used by lastsuccessful latch acquisition

ndash x$ksuprksllawer (v$process) ndash ldquowhererdquo that was used byprocess is being waiting for latch

ndash x$kslwscindx (v$latch_misses) ndash latch sleep statistics by

ldquowhererdquo

RDTEX | Latch internals | RUOUG Seminar

20

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2182

V$LATCH_MISSES

bull Based on X$KSLWSC - No[W]ait and [S]leep [C]ount stats byContext Documentation is rather obscure

bull The following counters are incremented on each latch sleep

sleep_count (ldquowhererdquo the latch was held)

wtr_slp_count (ldquowhererdquo the latch get was requested)

bull nwfail_count is incremented only for session which SID is equalto _LATCH_MISS_STAT_SID parameter

bull longhold_count is incremented for shared latches only when

someone held the latch at this ldquowhererdquo for the entire duration ofsleep

RDTEX | Latch internals | RUOUG Seminar

21

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2282

LATCH IS HOLDING BY PROCESS NOT SESSION

bull

RDTEX | Latch internals | RUOUG Seminar

983123983156983154983157983139983156 983147983155983157983152983154 983163

991270

983123983156983154983157983139983156 983147983155983148983148983137983163983147983155983148983148983156 98314798315598314898314898313798314898313798315698313114983133

983165

991270983165

983120983154983151983139983141983155983155 983142983145983160983141983140 983137983154983154983137983161

983158$983152983154983151983139983141983155983155 983085gt 983160$983147983155983157983152983154

983123983156983154983157983139983156 983147983155983148983148983156983163

991270

983165

L983145983155983156 983151983142 983137983148983148 983148983137983156983139983144983141983155

983158$983148983137983156983139983144 983085gt983160$983147983155983148983148983156

Each process has an array of references to the latches it is holding

22

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2382

KSLLA IS HIDDEN IN V$PROCESS

bull The process latching info is the kslla structure embedded in theprocess state object

bull Some kslla fields were externalized to SQL

bull kslla contains array of latches holding by process at each level andinformation about the state of any current attempt to acquire a

latch Not externalized directly

bull When you query v$latchholder x$ksuprlat Oracle scansv$processx$ksupr in order to find all the processes that are

holding the latchesbull Tanel Poder uses high frequency sampling of v$latchholder to

systematically troubleshoot latch contention

RDTEX | Latch internals | RUOUG Seminar

23

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2482

X$KSUPRKSLLA FIELDS INSTRUMENT THE LATCH GET

bull ksllalaq ndash address of latch acquiring Populated during immediateget (and spin before 11g)

bull ksllawat - latch being waited for This is v$processlatchwait

bull ksllawhy ndash ldquowhyrdquo for the latch being waited for

bull ksllawere ndash ldquowhererdquo for the latch being waited for

bull ksllalow ndash bit array of levels of currently holding latches

bull ksllaspn - latch this process is spinning on v$processlatchspinNot populated since 81

bull ksllaps - inter-process post statistics

RDTEX | Latch internals | RUOUG Seminar

24

Generic ldquolatch freerdquo and 27 latch specific ldquolatch free rdquo wait events areassociated with sleep only Spin is invisible to Oracle Wait Interface

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2582

THE LATCH STRUCTURE ndash KSLLT

RDTEX | Latch internals | RUOUG Seminar

struct ksllt

ltLatchgt

ldquowhererdquo and ldquowhyrdquo

Level latch class other attributes

Statistics

Latch wait list header

hellip

25

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2682

LATCH SIZE BY VERSION

nix 32bit nix 64bit Windows 32bit

734 92 - 120

806 104 - 104817 104 144 104

901 200 160

920 196 240 200

1010 256 208

1020 - 11203 100 160 104

RDTEX | Latch internals | RUOUG Seminar

x$ksmfsv ndash list of all fixed SGA variables983123E983116EC983124 DI983123983124I983118C983124 983147983155983149983142983155983155983145983162

F983122983119983117 983160$983147983155983149983142983155983158

983127HE983122E 983147983155983149983142983155983156983161983152 = 983147983155983148983148983156

Latch structure was bigger in 101 due to additional latch statistics

26

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2782

ORACLE LATCH IS NOT JUST A SINGLE MEMORY LOCATION

bull Before 11g Value of first latch byte (word for shared latches) wasused to determine latch state

0x00 ndash latch is free

0xFF ndash exclusive latch is busy Was 0x01 in Oracle 7

0x010x02hellip - shared latch holding by 12 hellip processes simultaneously

0x20000000 | pid - shared latch holding exclusively

bull In 11g first latch word shows the pid of the latch holder

0x00 ndash latch is free0x12 ndash Oracle process with pid 18 holds the exclusive latch

RDTEX | Latch internals | RUOUG Seminar

27

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2882

BUSY ORACLE 11G LATCH IN MEMORY

bull 112 Exclusive latch (32-bit platform)

bull 112 Shared latch EXCLUSIVE mode

RDTEX | Latch internals | RUOUG Seminar

SQLgt oradebug peek 200222A0 24

[200222A0 200222B8) = 00000016 00000001 000001D0 00000007

pid^ gets latch level

SQLgt oradebug peek 0xampladdr 24

[2000C814 2000C82C) = 2000001F 0000007C 0000007E 00000006 pid^ gets latch level

983139983151983150983156 983155983153983148983152983148983157983155 983150983151983148983151983143 9831409831419831499831519831350128

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2982

LATCH FIXED TABLES

bull Before Oracle 11g all the latches were externalized to SQL inv$latch (x$ksllt)

bull In 11g parent (fixed SGA) latches are externalized inv$latch_parent (x$kslltr_parent)

bull In 11g child (allocated on startup) latches are externalized in

v$latch_children (x$kslltr_children) bull Oracle 11g v$latch (x$kslltr) merges these data Beware select

from x$kslltr acquires all parent latches and their children

bull Only part of latch structure is visible in x$ksllthellip fixed tables ndash kslltwhr kslltwhy ndash ldquowhererdquo and ldquowhyrdquo latch was acquired last time

ndash kslltwgt hellip - statistics

ndash kslltlvl hellip - attributes

ndash kslltwtt ndash latch wait time

RDTEX | Latch internals | RUOUG Seminar

29

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3082

CUMULATIVE V$LATCH STATISTIC COUNTERS

RDTEX | Latch internals | RUOUG Seminar

Statistics x$ksllt When and how it increments

GETS kslltwgt ldquo++rdquo after wait mode latch get

MISSES kslltwff ldquo++rdquo after wait get if it was missed

SLEEPS kslltwsl ldquo+number_of_sleepsrdquo during get

SPIN_GETS ksllthst0 ldquo++rdquo if get was missed but not slept

WAIT_TIME kslltwtt ldquo+wait_timerdquo after latch get

IMMEDIATE_GETS kslltngt ldquo++rdquo after nowait mode latch get Is notincremented on miss

IMMEDIATE_MISSES kslltnfa ldquo++rdquo if nowait mode get was missed

30

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3182

MANY STATISTICS WERE LOST IN 102

In 102 Oracle made obsolete many useful latch statistics

Latch size was decreased and the following fields always contain 0now in v$latch

ndash v$latchwaiters_woken

ndash v$latchwaits_holding_latch

ndash v$latchsleep1hellipsleep10

ndash and many undocumented statistics

This was mentioned in 102 Reference

RDTEX | Latch internals | RUOUG Seminar

31

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3282

X$KSLLD - LATCH DESCRIPTORS

bull Flags and attributes describing each latch are stored in kslldt arrayof kslld structures

bull [K]ernel [S]ervice [L]atch [L]ock [D]escriptor

bull Some but not all of kslld fields are externalized via x$kslld andv$latchname

bull Located in the PGA

RDTEX | Latch internals | RUOUG Seminar

32

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382

LATCH ATTRIBUTES

bull Each latch have at least the following attributes in kslldt

ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared

ndash PAR Is the latch Solitary or Parent for the family of child latches

ndash G2C Can two child latches be simultaneously requested in wait mode

using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92

ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above

ndash Level 0-14 Used to prevent latch deadlocks

ndash Class 0-7 Spin and wait class assigned to the latch 92 and above

RDTEX | Latch internals | RUOUG Seminar

33

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482

LATCH ATTRIBUTES BY ORACLE VERSION

RDTEX | Latch internals | RUOUG Seminar

11203 553 154 72 - 6 93

Oracle Number of latches PAR G2C LNG UFS SHARED

7340 53 14 2 3 - -

8063 80 21 7 3 - 38174 152 48 19 4 - 9

9208 242 79 37 - - 19

10202 385 114 55 - 4 47

10203 388 117 58 - 4 48

10204 394 117 59 - 4 50

11106 496 145 67 - 6 81

11107 502 145 67 - 6 83

11201 535 149 70 - 6 86

11202 551 153 72 - 6 91

34

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582

LATCH LEVELS

bull Every latch has a level in the range 0-14 to avoid deadlocks

bull Process can request a latch X in wait mode after obtaining latch Y if and

only if level X gt level Y

bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y

bull If these rules are broken the Oracle process raises

ORA-600[504] - trying to obtain another latch at incompatible level Note 281041

ORA-600[525] - trying to obtain another latch at the same level Note 1388881

ORA-600[526] - latch child order violation Note 1388891

RDTEX | Latch internals | RUOUG Seminar

35

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682

PARENT AND CHILD LATCHES

bull Latch can be a parent child or solitary latch

bull Both parent and child latches share the same latch name

bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss

bull Child latches are allocated during instance startup

bull The parent latch can be gotten independently

bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s

RDTEX | Latch internals | RUOUG Seminar

(latch info) wait_event=0 bits=10

holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl

Context saved from call 0

state=busy

36

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782

LATCH TREES

ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches

Direct SGA access program output for 9206 instance with too smallshared pool

RDTEX | Latch internals | RUOUG Seminar

ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1

waiter ospid 129 sid 72 pid 45

holding a154b7120 library cache (157) level=5 child=17 kglupc child

waiter ospid 18255 sid 65 pid 930

waiter ospid 6690 sid 554 pid 1654

waiter ospid 4685 sid 879 pid 1034

hellip

waiter ospid 29749 sid 180 pid 155

holding a154b7db8 library cache (157) level=5 child=4 kglupc child

waiter ospid 13104 sid 281 pid 220

waiter ospid 24089 sid 565 pid 636

waiter ospid 25002 sid 621 pid 1481

waiter ospid 16930 sid 1046 pid 783hellip

37

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882

LATCH CLASSES

bull Classes with different spin and wait policies

bull Up to 8 classes By default all the latches belong to class 0

bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2

bull Latch can be assigned to class by its number For example

ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000

ndash _LATCH_CLASSES=46

bull Will be discussed later

RDTEX | Latch internals | RUOUG Seminar

SQLgt select from x$ksllclass

INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4

---- ------ ----- -------- ------ ------ ------ ------ ------

0 20000 0 1 8000 8000 8000 8000 8000

hellip

6 100 12 1000 2000 3000 4000 4000 4000

38

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982

LATCH ACQUISITION IN NO-WAIT MODE

bull kslgetl(laddr0whywhere) ndash Exclusive latch get

bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep

ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches

ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches

bull IMMEDIATE_ statistics are updated

bull Used when many candidate latches are available to bypass the

latch level rule or as part of CBC chain examination

RDTEX | Latch internals | RUOUG Seminar

sqlplus nolog demo_03sql39

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082

LATCH WAITS

RDTEX | Latch internals | RUOUG Seminar 40

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182

WAITING FOR THE LATCH

RDTEX | Latch internals | RUOUG Seminar

983123 G A

C983120983125 1 C983120983125 2

983120983154983151983139983141983155983155 A

983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)

983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144

983116983137983156983139983144

983120983154983151983139983141983155983155 B

41

RDTEX | L h i l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282

LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION

bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries

again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of

times

bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep

bull A sleep corresponds to a wait for the latch free wait event

bull Initially a process sleeps for 001 s This time is doubled in every

subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip

RDTEX | Latch internals | RUOUG Seminar

Still did not found any versionwhich used this

May be it was lt= Oracle 6

42

RDTEX | L t h i t l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382

LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81

Latch gets with wait (kslgetl(laddress1hellip))

ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times

ndash Sleep on latch free event with exponential backoff

ndash Repeat

RDTEX | Latch internals | RUOUG Seminar

43

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482

8I LATCH SPIN CODE FLOW USING DTRACE

RDTEX | Latch internals | RUOUG Seminar

bull hellip Event 10046 trace

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2

983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29

983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270

sqlplus nolog demo_04_8sql 29

44

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582

POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF

bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec

RDTEX | Latch internals | RUOUG Seminar

12 ]2 )1[(

minus= +wait N

timeout

45

bull Most waits were for nothing ndash latch already was free

bull Latch utilization can not be high

bull Unnecessary spins provoke CPU thrashing

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682

92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE

sqlplus nolog demo_05sql 45

RDTEX | Latch internals | RUOUG Seminar

kslgetl(0x50006318 1)

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt kslges(0x50006318 ) -wait latch get

-gt skgslsgts(0x50006318 ) -spin latch get

-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT

-gt kskthbwt(0x0)

-gt kslwlmod() - set up Wait List

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt skgpwwait -sleep latch getsemop(11 17-10 1)

983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140

46

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782

RELIABLE LATCH WAITS

bull Hidden latch wait revolution that we missed Latch relies on waitposting

bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout

bull Exclusive latch spins 20000 cycles by default

bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock

bull Latches assigned to non-default class wait until timeout

bull Version 90 was transient and used mix of backoff and wait posting

This is why _LATCH_WAIT_POSTING parameter disappeared in9i

bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin

RDTEX | Latch internals | RUOUG Seminar

47

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882

IMAGINE THE TIME MICROSCOPE

| a a s | a

Reality One million zoomed time

1 us 1 sec

1 ms 17 min1 sec 115 days

Light speed (300000 kmsec) Sonic speed (300 msec)

CPU tick (2 GHz) 00005 sec

Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec

Max spin time for shared latch (~220005 ticks) ~10 sec

Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec

Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec

OS context switch time (10us-1ms) 10sec-17min

48

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982

LATCH WAITS UNDER TIME MICROSCOPE

Old 8i latch algorithmSpin for latch during 5 seconds

Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min

hellipSpin again for 5 seconds

Sleep again for 23 days (2 sec) hellip

New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)

If spin not succeeded go to sleep until post

According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)

| |

49

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082

_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES

bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and

_LATCH_CLASS_0 parameters

bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000

bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter

bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch

50

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182

TIMED LATCH WAIT POSTING

bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel

bull This can lead to instance hang ndash process will never be woken up

bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout

bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem

bull Generic AIX ports always use timed latch wait posting

bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux

51

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282

FINE GRAIN LATCH TUNING

bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting

bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7

bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo

microseconds using pollsys() (not semtimedop()) system call

ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times

Loop up to ldquoSpinsrdquo cycles

Yield CPU using yield() (or sched_yield())

ndash Sleep for ldquoSleepXrdquo usecs

ndash Then spin again hellip

52

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382

TRACES OF NONDEFAULT CLASS LATCH

983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152

991270

9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000

991270 E983158983141983150983156 10046 983156983154983137983139983141

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482

PERFORMANCE OF LATCH CLASSES

Row cache objects latch contention testcase on ODA (24 SMT)54

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582

IS THE _SPIN_COUNT REALLY STATIC

bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch

waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used

ldquocache buffers chainrdquo latch contention

bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass

(_LATCH_CLASS_X) By default ndash 20000

bull Shared latch spin can be tuned by _SPIN_COUNT value By

default ndash 2000 However this is not trivial

55

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682

SHARED LATCH BEHAVE LIKE ENQUEUE

bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80

bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests

bull This blocking achieved by special 0x40000000 bit in the latch

value This bit is a flag that some incompatible latch acquisitionis in progress

bull Queued processes will be woken up one by one on latch release

bull X mode latch gets effectively serialize the shared latch

bull See my blog for description of experiments

56

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782

TRACE OF SHARED LATCH WAIT

Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the

latch wait list and then spins again

983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141

983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156

983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983147983155983148983159983148983149983151983140()

983147983155983148983155983147983143983155(098316050009BAC )

983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140

983155983141983149983151983152(27)

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882

SHARED LATCH ACQUISITION

bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned

bull X mode shared latch get spins by default up to 4000 cycles

bull S mode does not spin at all (or spins in unknown way)

S mode get X mode get

Held in S mode Compatible 2_SPIN_COUNT

Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT

58

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982

LATCH RELEASE

bull Free the latch ndash kslfre(laddr)

bull Oracle process releases the latch nonatomically

bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process

bull This requires less bus invalidation and ensures propagation of latch

release to other local cachesbull Not fair policy Spinners on the local CPU board have the

preference

bull Then process posts the first process in the list of waiters

59

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082

LATCH CONTENTION

RDTEX | Latch internals | RUOUG Seminar60

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182

LATCH CONTENTION

bull Occurs when a latch is required by several processes at thesame time

bull Mostly due to bugs or abnormal application behavior

bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp

bull To diagnose the latch contention we need to investigate thelatch statistics [8]

61

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282

DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS

Latch requests arrival rate

Immediate gets efficiency

Latch sleeps ratio

Latch wait time per second

Latch spin efficiencymisses

getsspin∆

∆= _σ

misses

sleepsK

∆=

time

gets

∆=λ

gets

misses

∆= ρ

time

timewait W

∆=

_

Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics

62

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382

SAMPLED AVERAGES

By sampling of x$ksupr we can estimate

ndash Length of latch wait queue

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS

ndash Number of spinning processes in 10g

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS

Sampling of v$lathholder estimates latch utilization

Using DTrace we can measure

ndash Latch acquisition and holding times

ndash Actual spin count

w L

s N

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482

DERIVED LATCH STATISTICS

Latch utilization (PASTA)

Average holding time

Length of latch wait list

Recurrent sleeps ratio

Latch acquisition time

time

timeholdinglatchU

∆=asymp

__ ρ

Re_100

___

questsGet

TimeSnap MissGet Pct S

lowast==

λ

W L =

κ

κ σ 1minus+

)(1

W N T saq += minusλ

64

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 14: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1482

ORADEBUG INTERNAL ORACLE DEBUGGER

RDTEX | Latch internals | RUOUG Seminar

bull oradebug call Allows us invoke any internal Oracle function manually

SQLgt oradebug peek 200222A0 24

[200222A0 200222B8) = 00000016 00000001 000001D0 00000007

SQLgt oradebug call kslgetl 0x5001A020 1 2 3Function returned 1

bull oradebug peek Examines contents of any memory location

bull oradebug poke Modifies memory

bull oradebug watch Watch a region of memory

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1582

LATCH CONTENTION TESTCASES

RDTEX | Latch internals | RUOUG Seminar

15

create table latch_contention as select from dualhellipfor i in 11000000loopselect dummy into a from latch_contention

end loop

bull Exclusive and shared latches behave differentlybull Contention for shared cache buffer chains latch arises when manysessions scan the same block concurrently and _fastpin_enable=0

bull To induce the contention for exclusive ldquorow cache objectsrdquo latch protectingdc_users row cache just add several dynamic RLS policies

exec dbms_rlsadd_policy(systemlatch_contentionrls_1 hellip

exec dbms_rlsadd_policy(systemlatch_contentionrls_2 helliphellip

bull These dummy policies generate predicates 1=1bull Many other contention scenarios possible

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313509831351

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1682

ROW CACHE LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

16

Elapsed and CPU times vs number of threadsOracle Database Appliance (24 SMT)

Latch wait

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1782

HOW ORACLEREQUESTS THE LATCH

RDTEX | Latch internals | RUOUG Seminar17

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1882

DTRACE REVEALS LATCH INTERFACE ROUTINES

Oracle calls the following functions to acquire the latch

ndash kslgetl(laddr wait why where) - get exclusive latch ndash kslg2c(l1l2trcwhy where) - get two excl child latches

ndash kslgetsl (laddrwaitwhywheremode) - get shared latch

bull 11g - ksl_get_shared_latch (hellip)

ndash kslg2cs(l1l2modetrcwhy where)) - get two shr child latches

ndash kslgpl(laddrcommentwhywhere) - get parent and all childs

ndash kslfre(laddr) - free the latch

Oracle give us possibility to do the same by oradebug call

RDTEX | Latch internals | RUOUG Seminar

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313501

983125983155983141 32983085983138983145983156 983152983151983154983156983155

64983138983145983156 O983154983137983139983148983141 983139983151983154983154983157983152983156983155 983156983144983141 983137983154983143983157983149983141983150983156983155 983151983142 983151983154983137983140983141983138983157983143 983139983137983148983148 983145983150 10204 983137983150983140 983137983138983151983158983141

18

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1982

ldquoWHERErdquo AND ldquoWHYrdquo ORACLE GETS THE LATCH

To request the latch Oracle kernel routines need

bull laddres ndash address of latch in SGAbull wait ndash flag for no-wait (0) or wait (1) latch acquisition

bull where ndash integer code for location from where the latch is acquired

Oracle lists possible latch ldquowhererdquo values in x$ksllwindxbull why - context of why the latch is acquiring at this ldquowhererdquo It

contains DBA for CBC latches (Tanel Poder) SGA chunk addressfor shared pool latch session address etchellip ldquoWhyrdquo meaning for

some ldquowhererdquo can be found in x$ksllwksllwlbl

bull mode ndash requesting state for shared lathes

ndash 8 SHARED mode 16 EXCLUSIVE mode

RDTEX | Latch internals | RUOUG Seminar

19

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2082

LIST OF LATCH ldquoWHERErdquo LOCATIONS

X$KSLLW ndash List of latch [W]here Part of v$latch_misses

ndash ksllwnam ndash Kernel function name(ldquokghupr1rdquo ldquoksqdelrdquo etc) ndash ksllwlbl ndash Description of ldquowhyrdquo context

Join this code on indx to

ndash x$ksuprlatksulawhr (v$latchholder)ndash ldquowhererdquo value that wasused by latch holder

ndash x$kslltkslltwhr (v$latch) ndash ldquowhererdquo that was used by lastsuccessful latch acquisition

ndash x$ksuprksllawer (v$process) ndash ldquowhererdquo that was used byprocess is being waiting for latch

ndash x$kslwscindx (v$latch_misses) ndash latch sleep statistics by

ldquowhererdquo

RDTEX | Latch internals | RUOUG Seminar

20

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2182

V$LATCH_MISSES

bull Based on X$KSLWSC - No[W]ait and [S]leep [C]ount stats byContext Documentation is rather obscure

bull The following counters are incremented on each latch sleep

sleep_count (ldquowhererdquo the latch was held)

wtr_slp_count (ldquowhererdquo the latch get was requested)

bull nwfail_count is incremented only for session which SID is equalto _LATCH_MISS_STAT_SID parameter

bull longhold_count is incremented for shared latches only when

someone held the latch at this ldquowhererdquo for the entire duration ofsleep

RDTEX | Latch internals | RUOUG Seminar

21

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2282

LATCH IS HOLDING BY PROCESS NOT SESSION

bull

RDTEX | Latch internals | RUOUG Seminar

983123983156983154983157983139983156 983147983155983157983152983154 983163

991270

983123983156983154983157983139983156 983147983155983148983148983137983163983147983155983148983148983156 98314798315598314898314898313798314898313798315698313114983133

983165

991270983165

983120983154983151983139983141983155983155 983142983145983160983141983140 983137983154983154983137983161

983158$983152983154983151983139983141983155983155 983085gt 983160$983147983155983157983152983154

983123983156983154983157983139983156 983147983155983148983148983156983163

991270

983165

L983145983155983156 983151983142 983137983148983148 983148983137983156983139983144983141983155

983158$983148983137983156983139983144 983085gt983160$983147983155983148983148983156

Each process has an array of references to the latches it is holding

22

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2382

KSLLA IS HIDDEN IN V$PROCESS

bull The process latching info is the kslla structure embedded in theprocess state object

bull Some kslla fields were externalized to SQL

bull kslla contains array of latches holding by process at each level andinformation about the state of any current attempt to acquire a

latch Not externalized directly

bull When you query v$latchholder x$ksuprlat Oracle scansv$processx$ksupr in order to find all the processes that are

holding the latchesbull Tanel Poder uses high frequency sampling of v$latchholder to

systematically troubleshoot latch contention

RDTEX | Latch internals | RUOUG Seminar

23

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2482

X$KSUPRKSLLA FIELDS INSTRUMENT THE LATCH GET

bull ksllalaq ndash address of latch acquiring Populated during immediateget (and spin before 11g)

bull ksllawat - latch being waited for This is v$processlatchwait

bull ksllawhy ndash ldquowhyrdquo for the latch being waited for

bull ksllawere ndash ldquowhererdquo for the latch being waited for

bull ksllalow ndash bit array of levels of currently holding latches

bull ksllaspn - latch this process is spinning on v$processlatchspinNot populated since 81

bull ksllaps - inter-process post statistics

RDTEX | Latch internals | RUOUG Seminar

24

Generic ldquolatch freerdquo and 27 latch specific ldquolatch free rdquo wait events areassociated with sleep only Spin is invisible to Oracle Wait Interface

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2582

THE LATCH STRUCTURE ndash KSLLT

RDTEX | Latch internals | RUOUG Seminar

struct ksllt

ltLatchgt

ldquowhererdquo and ldquowhyrdquo

Level latch class other attributes

Statistics

Latch wait list header

hellip

25

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2682

LATCH SIZE BY VERSION

nix 32bit nix 64bit Windows 32bit

734 92 - 120

806 104 - 104817 104 144 104

901 200 160

920 196 240 200

1010 256 208

1020 - 11203 100 160 104

RDTEX | Latch internals | RUOUG Seminar

x$ksmfsv ndash list of all fixed SGA variables983123E983116EC983124 DI983123983124I983118C983124 983147983155983149983142983155983155983145983162

F983122983119983117 983160$983147983155983149983142983155983158

983127HE983122E 983147983155983149983142983155983156983161983152 = 983147983155983148983148983156

Latch structure was bigger in 101 due to additional latch statistics

26

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2782

ORACLE LATCH IS NOT JUST A SINGLE MEMORY LOCATION

bull Before 11g Value of first latch byte (word for shared latches) wasused to determine latch state

0x00 ndash latch is free

0xFF ndash exclusive latch is busy Was 0x01 in Oracle 7

0x010x02hellip - shared latch holding by 12 hellip processes simultaneously

0x20000000 | pid - shared latch holding exclusively

bull In 11g first latch word shows the pid of the latch holder

0x00 ndash latch is free0x12 ndash Oracle process with pid 18 holds the exclusive latch

RDTEX | Latch internals | RUOUG Seminar

27

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2882

BUSY ORACLE 11G LATCH IN MEMORY

bull 112 Exclusive latch (32-bit platform)

bull 112 Shared latch EXCLUSIVE mode

RDTEX | Latch internals | RUOUG Seminar

SQLgt oradebug peek 200222A0 24

[200222A0 200222B8) = 00000016 00000001 000001D0 00000007

pid^ gets latch level

SQLgt oradebug peek 0xampladdr 24

[2000C814 2000C82C) = 2000001F 0000007C 0000007E 00000006 pid^ gets latch level

983139983151983150983156 983155983153983148983152983148983157983155 983150983151983148983151983143 9831409831419831499831519831350128

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2982

LATCH FIXED TABLES

bull Before Oracle 11g all the latches were externalized to SQL inv$latch (x$ksllt)

bull In 11g parent (fixed SGA) latches are externalized inv$latch_parent (x$kslltr_parent)

bull In 11g child (allocated on startup) latches are externalized in

v$latch_children (x$kslltr_children) bull Oracle 11g v$latch (x$kslltr) merges these data Beware select

from x$kslltr acquires all parent latches and their children

bull Only part of latch structure is visible in x$ksllthellip fixed tables ndash kslltwhr kslltwhy ndash ldquowhererdquo and ldquowhyrdquo latch was acquired last time

ndash kslltwgt hellip - statistics

ndash kslltlvl hellip - attributes

ndash kslltwtt ndash latch wait time

RDTEX | Latch internals | RUOUG Seminar

29

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3082

CUMULATIVE V$LATCH STATISTIC COUNTERS

RDTEX | Latch internals | RUOUG Seminar

Statistics x$ksllt When and how it increments

GETS kslltwgt ldquo++rdquo after wait mode latch get

MISSES kslltwff ldquo++rdquo after wait get if it was missed

SLEEPS kslltwsl ldquo+number_of_sleepsrdquo during get

SPIN_GETS ksllthst0 ldquo++rdquo if get was missed but not slept

WAIT_TIME kslltwtt ldquo+wait_timerdquo after latch get

IMMEDIATE_GETS kslltngt ldquo++rdquo after nowait mode latch get Is notincremented on miss

IMMEDIATE_MISSES kslltnfa ldquo++rdquo if nowait mode get was missed

30

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3182

MANY STATISTICS WERE LOST IN 102

In 102 Oracle made obsolete many useful latch statistics

Latch size was decreased and the following fields always contain 0now in v$latch

ndash v$latchwaiters_woken

ndash v$latchwaits_holding_latch

ndash v$latchsleep1hellipsleep10

ndash and many undocumented statistics

This was mentioned in 102 Reference

RDTEX | Latch internals | RUOUG Seminar

31

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3282

X$KSLLD - LATCH DESCRIPTORS

bull Flags and attributes describing each latch are stored in kslldt arrayof kslld structures

bull [K]ernel [S]ervice [L]atch [L]ock [D]escriptor

bull Some but not all of kslld fields are externalized via x$kslld andv$latchname

bull Located in the PGA

RDTEX | Latch internals | RUOUG Seminar

32

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382

LATCH ATTRIBUTES

bull Each latch have at least the following attributes in kslldt

ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared

ndash PAR Is the latch Solitary or Parent for the family of child latches

ndash G2C Can two child latches be simultaneously requested in wait mode

using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92

ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above

ndash Level 0-14 Used to prevent latch deadlocks

ndash Class 0-7 Spin and wait class assigned to the latch 92 and above

RDTEX | Latch internals | RUOUG Seminar

33

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482

LATCH ATTRIBUTES BY ORACLE VERSION

RDTEX | Latch internals | RUOUG Seminar

11203 553 154 72 - 6 93

Oracle Number of latches PAR G2C LNG UFS SHARED

7340 53 14 2 3 - -

8063 80 21 7 3 - 38174 152 48 19 4 - 9

9208 242 79 37 - - 19

10202 385 114 55 - 4 47

10203 388 117 58 - 4 48

10204 394 117 59 - 4 50

11106 496 145 67 - 6 81

11107 502 145 67 - 6 83

11201 535 149 70 - 6 86

11202 551 153 72 - 6 91

34

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582

LATCH LEVELS

bull Every latch has a level in the range 0-14 to avoid deadlocks

bull Process can request a latch X in wait mode after obtaining latch Y if and

only if level X gt level Y

bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y

bull If these rules are broken the Oracle process raises

ORA-600[504] - trying to obtain another latch at incompatible level Note 281041

ORA-600[525] - trying to obtain another latch at the same level Note 1388881

ORA-600[526] - latch child order violation Note 1388891

RDTEX | Latch internals | RUOUG Seminar

35

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682

PARENT AND CHILD LATCHES

bull Latch can be a parent child or solitary latch

bull Both parent and child latches share the same latch name

bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss

bull Child latches are allocated during instance startup

bull The parent latch can be gotten independently

bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s

RDTEX | Latch internals | RUOUG Seminar

(latch info) wait_event=0 bits=10

holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl

Context saved from call 0

state=busy

36

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782

LATCH TREES

ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches

Direct SGA access program output for 9206 instance with too smallshared pool

RDTEX | Latch internals | RUOUG Seminar

ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1

waiter ospid 129 sid 72 pid 45

holding a154b7120 library cache (157) level=5 child=17 kglupc child

waiter ospid 18255 sid 65 pid 930

waiter ospid 6690 sid 554 pid 1654

waiter ospid 4685 sid 879 pid 1034

hellip

waiter ospid 29749 sid 180 pid 155

holding a154b7db8 library cache (157) level=5 child=4 kglupc child

waiter ospid 13104 sid 281 pid 220

waiter ospid 24089 sid 565 pid 636

waiter ospid 25002 sid 621 pid 1481

waiter ospid 16930 sid 1046 pid 783hellip

37

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882

LATCH CLASSES

bull Classes with different spin and wait policies

bull Up to 8 classes By default all the latches belong to class 0

bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2

bull Latch can be assigned to class by its number For example

ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000

ndash _LATCH_CLASSES=46

bull Will be discussed later

RDTEX | Latch internals | RUOUG Seminar

SQLgt select from x$ksllclass

INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4

---- ------ ----- -------- ------ ------ ------ ------ ------

0 20000 0 1 8000 8000 8000 8000 8000

hellip

6 100 12 1000 2000 3000 4000 4000 4000

38

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982

LATCH ACQUISITION IN NO-WAIT MODE

bull kslgetl(laddr0whywhere) ndash Exclusive latch get

bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep

ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches

ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches

bull IMMEDIATE_ statistics are updated

bull Used when many candidate latches are available to bypass the

latch level rule or as part of CBC chain examination

RDTEX | Latch internals | RUOUG Seminar

sqlplus nolog demo_03sql39

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082

LATCH WAITS

RDTEX | Latch internals | RUOUG Seminar 40

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182

WAITING FOR THE LATCH

RDTEX | Latch internals | RUOUG Seminar

983123 G A

C983120983125 1 C983120983125 2

983120983154983151983139983141983155983155 A

983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)

983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144

983116983137983156983139983144

983120983154983151983139983141983155983155 B

41

RDTEX | L h i l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282

LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION

bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries

again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of

times

bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep

bull A sleep corresponds to a wait for the latch free wait event

bull Initially a process sleeps for 001 s This time is doubled in every

subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip

RDTEX | Latch internals | RUOUG Seminar

Still did not found any versionwhich used this

May be it was lt= Oracle 6

42

RDTEX | L t h i t l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382

LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81

Latch gets with wait (kslgetl(laddress1hellip))

ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times

ndash Sleep on latch free event with exponential backoff

ndash Repeat

RDTEX | Latch internals | RUOUG Seminar

43

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482

8I LATCH SPIN CODE FLOW USING DTRACE

RDTEX | Latch internals | RUOUG Seminar

bull hellip Event 10046 trace

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2

983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29

983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270

sqlplus nolog demo_04_8sql 29

44

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582

POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF

bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec

RDTEX | Latch internals | RUOUG Seminar

12 ]2 )1[(

minus= +wait N

timeout

45

bull Most waits were for nothing ndash latch already was free

bull Latch utilization can not be high

bull Unnecessary spins provoke CPU thrashing

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682

92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE

sqlplus nolog demo_05sql 45

RDTEX | Latch internals | RUOUG Seminar

kslgetl(0x50006318 1)

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt kslges(0x50006318 ) -wait latch get

-gt skgslsgts(0x50006318 ) -spin latch get

-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT

-gt kskthbwt(0x0)

-gt kslwlmod() - set up Wait List

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt skgpwwait -sleep latch getsemop(11 17-10 1)

983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140

46

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782

RELIABLE LATCH WAITS

bull Hidden latch wait revolution that we missed Latch relies on waitposting

bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout

bull Exclusive latch spins 20000 cycles by default

bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock

bull Latches assigned to non-default class wait until timeout

bull Version 90 was transient and used mix of backoff and wait posting

This is why _LATCH_WAIT_POSTING parameter disappeared in9i

bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin

RDTEX | Latch internals | RUOUG Seminar

47

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882

IMAGINE THE TIME MICROSCOPE

| a a s | a

Reality One million zoomed time

1 us 1 sec

1 ms 17 min1 sec 115 days

Light speed (300000 kmsec) Sonic speed (300 msec)

CPU tick (2 GHz) 00005 sec

Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec

Max spin time for shared latch (~220005 ticks) ~10 sec

Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec

Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec

OS context switch time (10us-1ms) 10sec-17min

48

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982

LATCH WAITS UNDER TIME MICROSCOPE

Old 8i latch algorithmSpin for latch during 5 seconds

Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min

hellipSpin again for 5 seconds

Sleep again for 23 days (2 sec) hellip

New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)

If spin not succeeded go to sleep until post

According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)

| |

49

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082

_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES

bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and

_LATCH_CLASS_0 parameters

bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000

bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter

bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch

50

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182

TIMED LATCH WAIT POSTING

bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel

bull This can lead to instance hang ndash process will never be woken up

bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout

bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem

bull Generic AIX ports always use timed latch wait posting

bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux

51

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282

FINE GRAIN LATCH TUNING

bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting

bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7

bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo

microseconds using pollsys() (not semtimedop()) system call

ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times

Loop up to ldquoSpinsrdquo cycles

Yield CPU using yield() (or sched_yield())

ndash Sleep for ldquoSleepXrdquo usecs

ndash Then spin again hellip

52

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382

TRACES OF NONDEFAULT CLASS LATCH

983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152

991270

9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000

991270 E983158983141983150983156 10046 983156983154983137983139983141

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482

PERFORMANCE OF LATCH CLASSES

Row cache objects latch contention testcase on ODA (24 SMT)54

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582

IS THE _SPIN_COUNT REALLY STATIC

bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch

waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used

ldquocache buffers chainrdquo latch contention

bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass

(_LATCH_CLASS_X) By default ndash 20000

bull Shared latch spin can be tuned by _SPIN_COUNT value By

default ndash 2000 However this is not trivial

55

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682

SHARED LATCH BEHAVE LIKE ENQUEUE

bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80

bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests

bull This blocking achieved by special 0x40000000 bit in the latch

value This bit is a flag that some incompatible latch acquisitionis in progress

bull Queued processes will be woken up one by one on latch release

bull X mode latch gets effectively serialize the shared latch

bull See my blog for description of experiments

56

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782

TRACE OF SHARED LATCH WAIT

Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the

latch wait list and then spins again

983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141

983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156

983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983147983155983148983159983148983149983151983140()

983147983155983148983155983147983143983155(098316050009BAC )

983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140

983155983141983149983151983152(27)

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882

SHARED LATCH ACQUISITION

bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned

bull X mode shared latch get spins by default up to 4000 cycles

bull S mode does not spin at all (or spins in unknown way)

S mode get X mode get

Held in S mode Compatible 2_SPIN_COUNT

Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT

58

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982

LATCH RELEASE

bull Free the latch ndash kslfre(laddr)

bull Oracle process releases the latch nonatomically

bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process

bull This requires less bus invalidation and ensures propagation of latch

release to other local cachesbull Not fair policy Spinners on the local CPU board have the

preference

bull Then process posts the first process in the list of waiters

59

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082

LATCH CONTENTION

RDTEX | Latch internals | RUOUG Seminar60

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182

LATCH CONTENTION

bull Occurs when a latch is required by several processes at thesame time

bull Mostly due to bugs or abnormal application behavior

bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp

bull To diagnose the latch contention we need to investigate thelatch statistics [8]

61

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282

DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS

Latch requests arrival rate

Immediate gets efficiency

Latch sleeps ratio

Latch wait time per second

Latch spin efficiencymisses

getsspin∆

∆= _σ

misses

sleepsK

∆=

time

gets

∆=λ

gets

misses

∆= ρ

time

timewait W

∆=

_

Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics

62

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382

SAMPLED AVERAGES

By sampling of x$ksupr we can estimate

ndash Length of latch wait queue

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS

ndash Number of spinning processes in 10g

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS

Sampling of v$lathholder estimates latch utilization

Using DTrace we can measure

ndash Latch acquisition and holding times

ndash Actual spin count

w L

s N

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482

DERIVED LATCH STATISTICS

Latch utilization (PASTA)

Average holding time

Length of latch wait list

Recurrent sleeps ratio

Latch acquisition time

time

timeholdinglatchU

∆=asymp

__ ρ

Re_100

___

questsGet

TimeSnap MissGet Pct S

lowast==

λ

W L =

κ

κ σ 1minus+

)(1

W N T saq += minusλ

64

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 15: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1582

LATCH CONTENTION TESTCASES

RDTEX | Latch internals | RUOUG Seminar

15

create table latch_contention as select from dualhellipfor i in 11000000loopselect dummy into a from latch_contention

end loop

bull Exclusive and shared latches behave differentlybull Contention for shared cache buffer chains latch arises when manysessions scan the same block concurrently and _fastpin_enable=0

bull To induce the contention for exclusive ldquorow cache objectsrdquo latch protectingdc_users row cache just add several dynamic RLS policies

exec dbms_rlsadd_policy(systemlatch_contentionrls_1 hellip

exec dbms_rlsadd_policy(systemlatch_contentionrls_2 helliphellip

bull These dummy policies generate predicates 1=1bull Many other contention scenarios possible

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313509831351

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1682

ROW CACHE LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

16

Elapsed and CPU times vs number of threadsOracle Database Appliance (24 SMT)

Latch wait

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1782

HOW ORACLEREQUESTS THE LATCH

RDTEX | Latch internals | RUOUG Seminar17

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1882

DTRACE REVEALS LATCH INTERFACE ROUTINES

Oracle calls the following functions to acquire the latch

ndash kslgetl(laddr wait why where) - get exclusive latch ndash kslg2c(l1l2trcwhy where) - get two excl child latches

ndash kslgetsl (laddrwaitwhywheremode) - get shared latch

bull 11g - ksl_get_shared_latch (hellip)

ndash kslg2cs(l1l2modetrcwhy where)) - get two shr child latches

ndash kslgpl(laddrcommentwhywhere) - get parent and all childs

ndash kslfre(laddr) - free the latch

Oracle give us possibility to do the same by oradebug call

RDTEX | Latch internals | RUOUG Seminar

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313501

983125983155983141 32983085983138983145983156 983152983151983154983156983155

64983138983145983156 O983154983137983139983148983141 983139983151983154983154983157983152983156983155 983156983144983141 983137983154983143983157983149983141983150983156983155 983151983142 983151983154983137983140983141983138983157983143 983139983137983148983148 983145983150 10204 983137983150983140 983137983138983151983158983141

18

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1982

ldquoWHERErdquo AND ldquoWHYrdquo ORACLE GETS THE LATCH

To request the latch Oracle kernel routines need

bull laddres ndash address of latch in SGAbull wait ndash flag for no-wait (0) or wait (1) latch acquisition

bull where ndash integer code for location from where the latch is acquired

Oracle lists possible latch ldquowhererdquo values in x$ksllwindxbull why - context of why the latch is acquiring at this ldquowhererdquo It

contains DBA for CBC latches (Tanel Poder) SGA chunk addressfor shared pool latch session address etchellip ldquoWhyrdquo meaning for

some ldquowhererdquo can be found in x$ksllwksllwlbl

bull mode ndash requesting state for shared lathes

ndash 8 SHARED mode 16 EXCLUSIVE mode

RDTEX | Latch internals | RUOUG Seminar

19

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2082

LIST OF LATCH ldquoWHERErdquo LOCATIONS

X$KSLLW ndash List of latch [W]here Part of v$latch_misses

ndash ksllwnam ndash Kernel function name(ldquokghupr1rdquo ldquoksqdelrdquo etc) ndash ksllwlbl ndash Description of ldquowhyrdquo context

Join this code on indx to

ndash x$ksuprlatksulawhr (v$latchholder)ndash ldquowhererdquo value that wasused by latch holder

ndash x$kslltkslltwhr (v$latch) ndash ldquowhererdquo that was used by lastsuccessful latch acquisition

ndash x$ksuprksllawer (v$process) ndash ldquowhererdquo that was used byprocess is being waiting for latch

ndash x$kslwscindx (v$latch_misses) ndash latch sleep statistics by

ldquowhererdquo

RDTEX | Latch internals | RUOUG Seminar

20

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2182

V$LATCH_MISSES

bull Based on X$KSLWSC - No[W]ait and [S]leep [C]ount stats byContext Documentation is rather obscure

bull The following counters are incremented on each latch sleep

sleep_count (ldquowhererdquo the latch was held)

wtr_slp_count (ldquowhererdquo the latch get was requested)

bull nwfail_count is incremented only for session which SID is equalto _LATCH_MISS_STAT_SID parameter

bull longhold_count is incremented for shared latches only when

someone held the latch at this ldquowhererdquo for the entire duration ofsleep

RDTEX | Latch internals | RUOUG Seminar

21

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2282

LATCH IS HOLDING BY PROCESS NOT SESSION

bull

RDTEX | Latch internals | RUOUG Seminar

983123983156983154983157983139983156 983147983155983157983152983154 983163

991270

983123983156983154983157983139983156 983147983155983148983148983137983163983147983155983148983148983156 98314798315598314898314898313798314898313798315698313114983133

983165

991270983165

983120983154983151983139983141983155983155 983142983145983160983141983140 983137983154983154983137983161

983158$983152983154983151983139983141983155983155 983085gt 983160$983147983155983157983152983154

983123983156983154983157983139983156 983147983155983148983148983156983163

991270

983165

L983145983155983156 983151983142 983137983148983148 983148983137983156983139983144983141983155

983158$983148983137983156983139983144 983085gt983160$983147983155983148983148983156

Each process has an array of references to the latches it is holding

22

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2382

KSLLA IS HIDDEN IN V$PROCESS

bull The process latching info is the kslla structure embedded in theprocess state object

bull Some kslla fields were externalized to SQL

bull kslla contains array of latches holding by process at each level andinformation about the state of any current attempt to acquire a

latch Not externalized directly

bull When you query v$latchholder x$ksuprlat Oracle scansv$processx$ksupr in order to find all the processes that are

holding the latchesbull Tanel Poder uses high frequency sampling of v$latchholder to

systematically troubleshoot latch contention

RDTEX | Latch internals | RUOUG Seminar

23

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2482

X$KSUPRKSLLA FIELDS INSTRUMENT THE LATCH GET

bull ksllalaq ndash address of latch acquiring Populated during immediateget (and spin before 11g)

bull ksllawat - latch being waited for This is v$processlatchwait

bull ksllawhy ndash ldquowhyrdquo for the latch being waited for

bull ksllawere ndash ldquowhererdquo for the latch being waited for

bull ksllalow ndash bit array of levels of currently holding latches

bull ksllaspn - latch this process is spinning on v$processlatchspinNot populated since 81

bull ksllaps - inter-process post statistics

RDTEX | Latch internals | RUOUG Seminar

24

Generic ldquolatch freerdquo and 27 latch specific ldquolatch free rdquo wait events areassociated with sleep only Spin is invisible to Oracle Wait Interface

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2582

THE LATCH STRUCTURE ndash KSLLT

RDTEX | Latch internals | RUOUG Seminar

struct ksllt

ltLatchgt

ldquowhererdquo and ldquowhyrdquo

Level latch class other attributes

Statistics

Latch wait list header

hellip

25

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2682

LATCH SIZE BY VERSION

nix 32bit nix 64bit Windows 32bit

734 92 - 120

806 104 - 104817 104 144 104

901 200 160

920 196 240 200

1010 256 208

1020 - 11203 100 160 104

RDTEX | Latch internals | RUOUG Seminar

x$ksmfsv ndash list of all fixed SGA variables983123E983116EC983124 DI983123983124I983118C983124 983147983155983149983142983155983155983145983162

F983122983119983117 983160$983147983155983149983142983155983158

983127HE983122E 983147983155983149983142983155983156983161983152 = 983147983155983148983148983156

Latch structure was bigger in 101 due to additional latch statistics

26

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2782

ORACLE LATCH IS NOT JUST A SINGLE MEMORY LOCATION

bull Before 11g Value of first latch byte (word for shared latches) wasused to determine latch state

0x00 ndash latch is free

0xFF ndash exclusive latch is busy Was 0x01 in Oracle 7

0x010x02hellip - shared latch holding by 12 hellip processes simultaneously

0x20000000 | pid - shared latch holding exclusively

bull In 11g first latch word shows the pid of the latch holder

0x00 ndash latch is free0x12 ndash Oracle process with pid 18 holds the exclusive latch

RDTEX | Latch internals | RUOUG Seminar

27

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2882

BUSY ORACLE 11G LATCH IN MEMORY

bull 112 Exclusive latch (32-bit platform)

bull 112 Shared latch EXCLUSIVE mode

RDTEX | Latch internals | RUOUG Seminar

SQLgt oradebug peek 200222A0 24

[200222A0 200222B8) = 00000016 00000001 000001D0 00000007

pid^ gets latch level

SQLgt oradebug peek 0xampladdr 24

[2000C814 2000C82C) = 2000001F 0000007C 0000007E 00000006 pid^ gets latch level

983139983151983150983156 983155983153983148983152983148983157983155 983150983151983148983151983143 9831409831419831499831519831350128

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2982

LATCH FIXED TABLES

bull Before Oracle 11g all the latches were externalized to SQL inv$latch (x$ksllt)

bull In 11g parent (fixed SGA) latches are externalized inv$latch_parent (x$kslltr_parent)

bull In 11g child (allocated on startup) latches are externalized in

v$latch_children (x$kslltr_children) bull Oracle 11g v$latch (x$kslltr) merges these data Beware select

from x$kslltr acquires all parent latches and their children

bull Only part of latch structure is visible in x$ksllthellip fixed tables ndash kslltwhr kslltwhy ndash ldquowhererdquo and ldquowhyrdquo latch was acquired last time

ndash kslltwgt hellip - statistics

ndash kslltlvl hellip - attributes

ndash kslltwtt ndash latch wait time

RDTEX | Latch internals | RUOUG Seminar

29

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3082

CUMULATIVE V$LATCH STATISTIC COUNTERS

RDTEX | Latch internals | RUOUG Seminar

Statistics x$ksllt When and how it increments

GETS kslltwgt ldquo++rdquo after wait mode latch get

MISSES kslltwff ldquo++rdquo after wait get if it was missed

SLEEPS kslltwsl ldquo+number_of_sleepsrdquo during get

SPIN_GETS ksllthst0 ldquo++rdquo if get was missed but not slept

WAIT_TIME kslltwtt ldquo+wait_timerdquo after latch get

IMMEDIATE_GETS kslltngt ldquo++rdquo after nowait mode latch get Is notincremented on miss

IMMEDIATE_MISSES kslltnfa ldquo++rdquo if nowait mode get was missed

30

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3182

MANY STATISTICS WERE LOST IN 102

In 102 Oracle made obsolete many useful latch statistics

Latch size was decreased and the following fields always contain 0now in v$latch

ndash v$latchwaiters_woken

ndash v$latchwaits_holding_latch

ndash v$latchsleep1hellipsleep10

ndash and many undocumented statistics

This was mentioned in 102 Reference

RDTEX | Latch internals | RUOUG Seminar

31

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3282

X$KSLLD - LATCH DESCRIPTORS

bull Flags and attributes describing each latch are stored in kslldt arrayof kslld structures

bull [K]ernel [S]ervice [L]atch [L]ock [D]escriptor

bull Some but not all of kslld fields are externalized via x$kslld andv$latchname

bull Located in the PGA

RDTEX | Latch internals | RUOUG Seminar

32

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382

LATCH ATTRIBUTES

bull Each latch have at least the following attributes in kslldt

ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared

ndash PAR Is the latch Solitary or Parent for the family of child latches

ndash G2C Can two child latches be simultaneously requested in wait mode

using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92

ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above

ndash Level 0-14 Used to prevent latch deadlocks

ndash Class 0-7 Spin and wait class assigned to the latch 92 and above

RDTEX | Latch internals | RUOUG Seminar

33

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482

LATCH ATTRIBUTES BY ORACLE VERSION

RDTEX | Latch internals | RUOUG Seminar

11203 553 154 72 - 6 93

Oracle Number of latches PAR G2C LNG UFS SHARED

7340 53 14 2 3 - -

8063 80 21 7 3 - 38174 152 48 19 4 - 9

9208 242 79 37 - - 19

10202 385 114 55 - 4 47

10203 388 117 58 - 4 48

10204 394 117 59 - 4 50

11106 496 145 67 - 6 81

11107 502 145 67 - 6 83

11201 535 149 70 - 6 86

11202 551 153 72 - 6 91

34

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582

LATCH LEVELS

bull Every latch has a level in the range 0-14 to avoid deadlocks

bull Process can request a latch X in wait mode after obtaining latch Y if and

only if level X gt level Y

bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y

bull If these rules are broken the Oracle process raises

ORA-600[504] - trying to obtain another latch at incompatible level Note 281041

ORA-600[525] - trying to obtain another latch at the same level Note 1388881

ORA-600[526] - latch child order violation Note 1388891

RDTEX | Latch internals | RUOUG Seminar

35

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682

PARENT AND CHILD LATCHES

bull Latch can be a parent child or solitary latch

bull Both parent and child latches share the same latch name

bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss

bull Child latches are allocated during instance startup

bull The parent latch can be gotten independently

bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s

RDTEX | Latch internals | RUOUG Seminar

(latch info) wait_event=0 bits=10

holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl

Context saved from call 0

state=busy

36

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782

LATCH TREES

ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches

Direct SGA access program output for 9206 instance with too smallshared pool

RDTEX | Latch internals | RUOUG Seminar

ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1

waiter ospid 129 sid 72 pid 45

holding a154b7120 library cache (157) level=5 child=17 kglupc child

waiter ospid 18255 sid 65 pid 930

waiter ospid 6690 sid 554 pid 1654

waiter ospid 4685 sid 879 pid 1034

hellip

waiter ospid 29749 sid 180 pid 155

holding a154b7db8 library cache (157) level=5 child=4 kglupc child

waiter ospid 13104 sid 281 pid 220

waiter ospid 24089 sid 565 pid 636

waiter ospid 25002 sid 621 pid 1481

waiter ospid 16930 sid 1046 pid 783hellip

37

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882

LATCH CLASSES

bull Classes with different spin and wait policies

bull Up to 8 classes By default all the latches belong to class 0

bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2

bull Latch can be assigned to class by its number For example

ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000

ndash _LATCH_CLASSES=46

bull Will be discussed later

RDTEX | Latch internals | RUOUG Seminar

SQLgt select from x$ksllclass

INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4

---- ------ ----- -------- ------ ------ ------ ------ ------

0 20000 0 1 8000 8000 8000 8000 8000

hellip

6 100 12 1000 2000 3000 4000 4000 4000

38

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982

LATCH ACQUISITION IN NO-WAIT MODE

bull kslgetl(laddr0whywhere) ndash Exclusive latch get

bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep

ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches

ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches

bull IMMEDIATE_ statistics are updated

bull Used when many candidate latches are available to bypass the

latch level rule or as part of CBC chain examination

RDTEX | Latch internals | RUOUG Seminar

sqlplus nolog demo_03sql39

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082

LATCH WAITS

RDTEX | Latch internals | RUOUG Seminar 40

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182

WAITING FOR THE LATCH

RDTEX | Latch internals | RUOUG Seminar

983123 G A

C983120983125 1 C983120983125 2

983120983154983151983139983141983155983155 A

983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)

983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144

983116983137983156983139983144

983120983154983151983139983141983155983155 B

41

RDTEX | L h i l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282

LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION

bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries

again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of

times

bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep

bull A sleep corresponds to a wait for the latch free wait event

bull Initially a process sleeps for 001 s This time is doubled in every

subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip

RDTEX | Latch internals | RUOUG Seminar

Still did not found any versionwhich used this

May be it was lt= Oracle 6

42

RDTEX | L t h i t l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382

LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81

Latch gets with wait (kslgetl(laddress1hellip))

ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times

ndash Sleep on latch free event with exponential backoff

ndash Repeat

RDTEX | Latch internals | RUOUG Seminar

43

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482

8I LATCH SPIN CODE FLOW USING DTRACE

RDTEX | Latch internals | RUOUG Seminar

bull hellip Event 10046 trace

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2

983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29

983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270

sqlplus nolog demo_04_8sql 29

44

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582

POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF

bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec

RDTEX | Latch internals | RUOUG Seminar

12 ]2 )1[(

minus= +wait N

timeout

45

bull Most waits were for nothing ndash latch already was free

bull Latch utilization can not be high

bull Unnecessary spins provoke CPU thrashing

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682

92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE

sqlplus nolog demo_05sql 45

RDTEX | Latch internals | RUOUG Seminar

kslgetl(0x50006318 1)

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt kslges(0x50006318 ) -wait latch get

-gt skgslsgts(0x50006318 ) -spin latch get

-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT

-gt kskthbwt(0x0)

-gt kslwlmod() - set up Wait List

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt skgpwwait -sleep latch getsemop(11 17-10 1)

983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140

46

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782

RELIABLE LATCH WAITS

bull Hidden latch wait revolution that we missed Latch relies on waitposting

bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout

bull Exclusive latch spins 20000 cycles by default

bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock

bull Latches assigned to non-default class wait until timeout

bull Version 90 was transient and used mix of backoff and wait posting

This is why _LATCH_WAIT_POSTING parameter disappeared in9i

bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin

RDTEX | Latch internals | RUOUG Seminar

47

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882

IMAGINE THE TIME MICROSCOPE

| a a s | a

Reality One million zoomed time

1 us 1 sec

1 ms 17 min1 sec 115 days

Light speed (300000 kmsec) Sonic speed (300 msec)

CPU tick (2 GHz) 00005 sec

Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec

Max spin time for shared latch (~220005 ticks) ~10 sec

Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec

Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec

OS context switch time (10us-1ms) 10sec-17min

48

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982

LATCH WAITS UNDER TIME MICROSCOPE

Old 8i latch algorithmSpin for latch during 5 seconds

Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min

hellipSpin again for 5 seconds

Sleep again for 23 days (2 sec) hellip

New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)

If spin not succeeded go to sleep until post

According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)

| |

49

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082

_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES

bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and

_LATCH_CLASS_0 parameters

bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000

bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter

bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch

50

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182

TIMED LATCH WAIT POSTING

bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel

bull This can lead to instance hang ndash process will never be woken up

bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout

bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem

bull Generic AIX ports always use timed latch wait posting

bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux

51

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282

FINE GRAIN LATCH TUNING

bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting

bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7

bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo

microseconds using pollsys() (not semtimedop()) system call

ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times

Loop up to ldquoSpinsrdquo cycles

Yield CPU using yield() (or sched_yield())

ndash Sleep for ldquoSleepXrdquo usecs

ndash Then spin again hellip

52

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382

TRACES OF NONDEFAULT CLASS LATCH

983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152

991270

9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000

991270 E983158983141983150983156 10046 983156983154983137983139983141

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482

PERFORMANCE OF LATCH CLASSES

Row cache objects latch contention testcase on ODA (24 SMT)54

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582

IS THE _SPIN_COUNT REALLY STATIC

bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch

waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used

ldquocache buffers chainrdquo latch contention

bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass

(_LATCH_CLASS_X) By default ndash 20000

bull Shared latch spin can be tuned by _SPIN_COUNT value By

default ndash 2000 However this is not trivial

55

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682

SHARED LATCH BEHAVE LIKE ENQUEUE

bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80

bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests

bull This blocking achieved by special 0x40000000 bit in the latch

value This bit is a flag that some incompatible latch acquisitionis in progress

bull Queued processes will be woken up one by one on latch release

bull X mode latch gets effectively serialize the shared latch

bull See my blog for description of experiments

56

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782

TRACE OF SHARED LATCH WAIT

Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the

latch wait list and then spins again

983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141

983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156

983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983147983155983148983159983148983149983151983140()

983147983155983148983155983147983143983155(098316050009BAC )

983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140

983155983141983149983151983152(27)

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882

SHARED LATCH ACQUISITION

bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned

bull X mode shared latch get spins by default up to 4000 cycles

bull S mode does not spin at all (or spins in unknown way)

S mode get X mode get

Held in S mode Compatible 2_SPIN_COUNT

Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT

58

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982

LATCH RELEASE

bull Free the latch ndash kslfre(laddr)

bull Oracle process releases the latch nonatomically

bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process

bull This requires less bus invalidation and ensures propagation of latch

release to other local cachesbull Not fair policy Spinners on the local CPU board have the

preference

bull Then process posts the first process in the list of waiters

59

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082

LATCH CONTENTION

RDTEX | Latch internals | RUOUG Seminar60

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182

LATCH CONTENTION

bull Occurs when a latch is required by several processes at thesame time

bull Mostly due to bugs or abnormal application behavior

bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp

bull To diagnose the latch contention we need to investigate thelatch statistics [8]

61

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282

DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS

Latch requests arrival rate

Immediate gets efficiency

Latch sleeps ratio

Latch wait time per second

Latch spin efficiencymisses

getsspin∆

∆= _σ

misses

sleepsK

∆=

time

gets

∆=λ

gets

misses

∆= ρ

time

timewait W

∆=

_

Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics

62

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382

SAMPLED AVERAGES

By sampling of x$ksupr we can estimate

ndash Length of latch wait queue

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS

ndash Number of spinning processes in 10g

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS

Sampling of v$lathholder estimates latch utilization

Using DTrace we can measure

ndash Latch acquisition and holding times

ndash Actual spin count

w L

s N

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482

DERIVED LATCH STATISTICS

Latch utilization (PASTA)

Average holding time

Length of latch wait list

Recurrent sleeps ratio

Latch acquisition time

time

timeholdinglatchU

∆=asymp

__ ρ

Re_100

___

questsGet

TimeSnap MissGet Pct S

lowast==

λ

W L =

κ

κ σ 1minus+

)(1

W N T saq += minusλ

64

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 16: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1682

ROW CACHE LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

16

Elapsed and CPU times vs number of threadsOracle Database Appliance (24 SMT)

Latch wait

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1782

HOW ORACLEREQUESTS THE LATCH

RDTEX | Latch internals | RUOUG Seminar17

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1882

DTRACE REVEALS LATCH INTERFACE ROUTINES

Oracle calls the following functions to acquire the latch

ndash kslgetl(laddr wait why where) - get exclusive latch ndash kslg2c(l1l2trcwhy where) - get two excl child latches

ndash kslgetsl (laddrwaitwhywheremode) - get shared latch

bull 11g - ksl_get_shared_latch (hellip)

ndash kslg2cs(l1l2modetrcwhy where)) - get two shr child latches

ndash kslgpl(laddrcommentwhywhere) - get parent and all childs

ndash kslfre(laddr) - free the latch

Oracle give us possibility to do the same by oradebug call

RDTEX | Latch internals | RUOUG Seminar

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313501

983125983155983141 32983085983138983145983156 983152983151983154983156983155

64983138983145983156 O983154983137983139983148983141 983139983151983154983154983157983152983156983155 983156983144983141 983137983154983143983157983149983141983150983156983155 983151983142 983151983154983137983140983141983138983157983143 983139983137983148983148 983145983150 10204 983137983150983140 983137983138983151983158983141

18

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1982

ldquoWHERErdquo AND ldquoWHYrdquo ORACLE GETS THE LATCH

To request the latch Oracle kernel routines need

bull laddres ndash address of latch in SGAbull wait ndash flag for no-wait (0) or wait (1) latch acquisition

bull where ndash integer code for location from where the latch is acquired

Oracle lists possible latch ldquowhererdquo values in x$ksllwindxbull why - context of why the latch is acquiring at this ldquowhererdquo It

contains DBA for CBC latches (Tanel Poder) SGA chunk addressfor shared pool latch session address etchellip ldquoWhyrdquo meaning for

some ldquowhererdquo can be found in x$ksllwksllwlbl

bull mode ndash requesting state for shared lathes

ndash 8 SHARED mode 16 EXCLUSIVE mode

RDTEX | Latch internals | RUOUG Seminar

19

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2082

LIST OF LATCH ldquoWHERErdquo LOCATIONS

X$KSLLW ndash List of latch [W]here Part of v$latch_misses

ndash ksllwnam ndash Kernel function name(ldquokghupr1rdquo ldquoksqdelrdquo etc) ndash ksllwlbl ndash Description of ldquowhyrdquo context

Join this code on indx to

ndash x$ksuprlatksulawhr (v$latchholder)ndash ldquowhererdquo value that wasused by latch holder

ndash x$kslltkslltwhr (v$latch) ndash ldquowhererdquo that was used by lastsuccessful latch acquisition

ndash x$ksuprksllawer (v$process) ndash ldquowhererdquo that was used byprocess is being waiting for latch

ndash x$kslwscindx (v$latch_misses) ndash latch sleep statistics by

ldquowhererdquo

RDTEX | Latch internals | RUOUG Seminar

20

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2182

V$LATCH_MISSES

bull Based on X$KSLWSC - No[W]ait and [S]leep [C]ount stats byContext Documentation is rather obscure

bull The following counters are incremented on each latch sleep

sleep_count (ldquowhererdquo the latch was held)

wtr_slp_count (ldquowhererdquo the latch get was requested)

bull nwfail_count is incremented only for session which SID is equalto _LATCH_MISS_STAT_SID parameter

bull longhold_count is incremented for shared latches only when

someone held the latch at this ldquowhererdquo for the entire duration ofsleep

RDTEX | Latch internals | RUOUG Seminar

21

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2282

LATCH IS HOLDING BY PROCESS NOT SESSION

bull

RDTEX | Latch internals | RUOUG Seminar

983123983156983154983157983139983156 983147983155983157983152983154 983163

991270

983123983156983154983157983139983156 983147983155983148983148983137983163983147983155983148983148983156 98314798315598314898314898313798314898313798315698313114983133

983165

991270983165

983120983154983151983139983141983155983155 983142983145983160983141983140 983137983154983154983137983161

983158$983152983154983151983139983141983155983155 983085gt 983160$983147983155983157983152983154

983123983156983154983157983139983156 983147983155983148983148983156983163

991270

983165

L983145983155983156 983151983142 983137983148983148 983148983137983156983139983144983141983155

983158$983148983137983156983139983144 983085gt983160$983147983155983148983148983156

Each process has an array of references to the latches it is holding

22

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2382

KSLLA IS HIDDEN IN V$PROCESS

bull The process latching info is the kslla structure embedded in theprocess state object

bull Some kslla fields were externalized to SQL

bull kslla contains array of latches holding by process at each level andinformation about the state of any current attempt to acquire a

latch Not externalized directly

bull When you query v$latchholder x$ksuprlat Oracle scansv$processx$ksupr in order to find all the processes that are

holding the latchesbull Tanel Poder uses high frequency sampling of v$latchholder to

systematically troubleshoot latch contention

RDTEX | Latch internals | RUOUG Seminar

23

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2482

X$KSUPRKSLLA FIELDS INSTRUMENT THE LATCH GET

bull ksllalaq ndash address of latch acquiring Populated during immediateget (and spin before 11g)

bull ksllawat - latch being waited for This is v$processlatchwait

bull ksllawhy ndash ldquowhyrdquo for the latch being waited for

bull ksllawere ndash ldquowhererdquo for the latch being waited for

bull ksllalow ndash bit array of levels of currently holding latches

bull ksllaspn - latch this process is spinning on v$processlatchspinNot populated since 81

bull ksllaps - inter-process post statistics

RDTEX | Latch internals | RUOUG Seminar

24

Generic ldquolatch freerdquo and 27 latch specific ldquolatch free rdquo wait events areassociated with sleep only Spin is invisible to Oracle Wait Interface

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2582

THE LATCH STRUCTURE ndash KSLLT

RDTEX | Latch internals | RUOUG Seminar

struct ksllt

ltLatchgt

ldquowhererdquo and ldquowhyrdquo

Level latch class other attributes

Statistics

Latch wait list header

hellip

25

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2682

LATCH SIZE BY VERSION

nix 32bit nix 64bit Windows 32bit

734 92 - 120

806 104 - 104817 104 144 104

901 200 160

920 196 240 200

1010 256 208

1020 - 11203 100 160 104

RDTEX | Latch internals | RUOUG Seminar

x$ksmfsv ndash list of all fixed SGA variables983123E983116EC983124 DI983123983124I983118C983124 983147983155983149983142983155983155983145983162

F983122983119983117 983160$983147983155983149983142983155983158

983127HE983122E 983147983155983149983142983155983156983161983152 = 983147983155983148983148983156

Latch structure was bigger in 101 due to additional latch statistics

26

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2782

ORACLE LATCH IS NOT JUST A SINGLE MEMORY LOCATION

bull Before 11g Value of first latch byte (word for shared latches) wasused to determine latch state

0x00 ndash latch is free

0xFF ndash exclusive latch is busy Was 0x01 in Oracle 7

0x010x02hellip - shared latch holding by 12 hellip processes simultaneously

0x20000000 | pid - shared latch holding exclusively

bull In 11g first latch word shows the pid of the latch holder

0x00 ndash latch is free0x12 ndash Oracle process with pid 18 holds the exclusive latch

RDTEX | Latch internals | RUOUG Seminar

27

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2882

BUSY ORACLE 11G LATCH IN MEMORY

bull 112 Exclusive latch (32-bit platform)

bull 112 Shared latch EXCLUSIVE mode

RDTEX | Latch internals | RUOUG Seminar

SQLgt oradebug peek 200222A0 24

[200222A0 200222B8) = 00000016 00000001 000001D0 00000007

pid^ gets latch level

SQLgt oradebug peek 0xampladdr 24

[2000C814 2000C82C) = 2000001F 0000007C 0000007E 00000006 pid^ gets latch level

983139983151983150983156 983155983153983148983152983148983157983155 983150983151983148983151983143 9831409831419831499831519831350128

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2982

LATCH FIXED TABLES

bull Before Oracle 11g all the latches were externalized to SQL inv$latch (x$ksllt)

bull In 11g parent (fixed SGA) latches are externalized inv$latch_parent (x$kslltr_parent)

bull In 11g child (allocated on startup) latches are externalized in

v$latch_children (x$kslltr_children) bull Oracle 11g v$latch (x$kslltr) merges these data Beware select

from x$kslltr acquires all parent latches and their children

bull Only part of latch structure is visible in x$ksllthellip fixed tables ndash kslltwhr kslltwhy ndash ldquowhererdquo and ldquowhyrdquo latch was acquired last time

ndash kslltwgt hellip - statistics

ndash kslltlvl hellip - attributes

ndash kslltwtt ndash latch wait time

RDTEX | Latch internals | RUOUG Seminar

29

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3082

CUMULATIVE V$LATCH STATISTIC COUNTERS

RDTEX | Latch internals | RUOUG Seminar

Statistics x$ksllt When and how it increments

GETS kslltwgt ldquo++rdquo after wait mode latch get

MISSES kslltwff ldquo++rdquo after wait get if it was missed

SLEEPS kslltwsl ldquo+number_of_sleepsrdquo during get

SPIN_GETS ksllthst0 ldquo++rdquo if get was missed but not slept

WAIT_TIME kslltwtt ldquo+wait_timerdquo after latch get

IMMEDIATE_GETS kslltngt ldquo++rdquo after nowait mode latch get Is notincremented on miss

IMMEDIATE_MISSES kslltnfa ldquo++rdquo if nowait mode get was missed

30

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3182

MANY STATISTICS WERE LOST IN 102

In 102 Oracle made obsolete many useful latch statistics

Latch size was decreased and the following fields always contain 0now in v$latch

ndash v$latchwaiters_woken

ndash v$latchwaits_holding_latch

ndash v$latchsleep1hellipsleep10

ndash and many undocumented statistics

This was mentioned in 102 Reference

RDTEX | Latch internals | RUOUG Seminar

31

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3282

X$KSLLD - LATCH DESCRIPTORS

bull Flags and attributes describing each latch are stored in kslldt arrayof kslld structures

bull [K]ernel [S]ervice [L]atch [L]ock [D]escriptor

bull Some but not all of kslld fields are externalized via x$kslld andv$latchname

bull Located in the PGA

RDTEX | Latch internals | RUOUG Seminar

32

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382

LATCH ATTRIBUTES

bull Each latch have at least the following attributes in kslldt

ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared

ndash PAR Is the latch Solitary or Parent for the family of child latches

ndash G2C Can two child latches be simultaneously requested in wait mode

using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92

ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above

ndash Level 0-14 Used to prevent latch deadlocks

ndash Class 0-7 Spin and wait class assigned to the latch 92 and above

RDTEX | Latch internals | RUOUG Seminar

33

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482

LATCH ATTRIBUTES BY ORACLE VERSION

RDTEX | Latch internals | RUOUG Seminar

11203 553 154 72 - 6 93

Oracle Number of latches PAR G2C LNG UFS SHARED

7340 53 14 2 3 - -

8063 80 21 7 3 - 38174 152 48 19 4 - 9

9208 242 79 37 - - 19

10202 385 114 55 - 4 47

10203 388 117 58 - 4 48

10204 394 117 59 - 4 50

11106 496 145 67 - 6 81

11107 502 145 67 - 6 83

11201 535 149 70 - 6 86

11202 551 153 72 - 6 91

34

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582

LATCH LEVELS

bull Every latch has a level in the range 0-14 to avoid deadlocks

bull Process can request a latch X in wait mode after obtaining latch Y if and

only if level X gt level Y

bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y

bull If these rules are broken the Oracle process raises

ORA-600[504] - trying to obtain another latch at incompatible level Note 281041

ORA-600[525] - trying to obtain another latch at the same level Note 1388881

ORA-600[526] - latch child order violation Note 1388891

RDTEX | Latch internals | RUOUG Seminar

35

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682

PARENT AND CHILD LATCHES

bull Latch can be a parent child or solitary latch

bull Both parent and child latches share the same latch name

bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss

bull Child latches are allocated during instance startup

bull The parent latch can be gotten independently

bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s

RDTEX | Latch internals | RUOUG Seminar

(latch info) wait_event=0 bits=10

holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl

Context saved from call 0

state=busy

36

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782

LATCH TREES

ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches

Direct SGA access program output for 9206 instance with too smallshared pool

RDTEX | Latch internals | RUOUG Seminar

ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1

waiter ospid 129 sid 72 pid 45

holding a154b7120 library cache (157) level=5 child=17 kglupc child

waiter ospid 18255 sid 65 pid 930

waiter ospid 6690 sid 554 pid 1654

waiter ospid 4685 sid 879 pid 1034

hellip

waiter ospid 29749 sid 180 pid 155

holding a154b7db8 library cache (157) level=5 child=4 kglupc child

waiter ospid 13104 sid 281 pid 220

waiter ospid 24089 sid 565 pid 636

waiter ospid 25002 sid 621 pid 1481

waiter ospid 16930 sid 1046 pid 783hellip

37

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882

LATCH CLASSES

bull Classes with different spin and wait policies

bull Up to 8 classes By default all the latches belong to class 0

bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2

bull Latch can be assigned to class by its number For example

ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000

ndash _LATCH_CLASSES=46

bull Will be discussed later

RDTEX | Latch internals | RUOUG Seminar

SQLgt select from x$ksllclass

INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4

---- ------ ----- -------- ------ ------ ------ ------ ------

0 20000 0 1 8000 8000 8000 8000 8000

hellip

6 100 12 1000 2000 3000 4000 4000 4000

38

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982

LATCH ACQUISITION IN NO-WAIT MODE

bull kslgetl(laddr0whywhere) ndash Exclusive latch get

bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep

ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches

ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches

bull IMMEDIATE_ statistics are updated

bull Used when many candidate latches are available to bypass the

latch level rule or as part of CBC chain examination

RDTEX | Latch internals | RUOUG Seminar

sqlplus nolog demo_03sql39

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082

LATCH WAITS

RDTEX | Latch internals | RUOUG Seminar 40

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182

WAITING FOR THE LATCH

RDTEX | Latch internals | RUOUG Seminar

983123 G A

C983120983125 1 C983120983125 2

983120983154983151983139983141983155983155 A

983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)

983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144

983116983137983156983139983144

983120983154983151983139983141983155983155 B

41

RDTEX | L h i l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282

LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION

bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries

again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of

times

bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep

bull A sleep corresponds to a wait for the latch free wait event

bull Initially a process sleeps for 001 s This time is doubled in every

subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip

RDTEX | Latch internals | RUOUG Seminar

Still did not found any versionwhich used this

May be it was lt= Oracle 6

42

RDTEX | L t h i t l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382

LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81

Latch gets with wait (kslgetl(laddress1hellip))

ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times

ndash Sleep on latch free event with exponential backoff

ndash Repeat

RDTEX | Latch internals | RUOUG Seminar

43

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482

8I LATCH SPIN CODE FLOW USING DTRACE

RDTEX | Latch internals | RUOUG Seminar

bull hellip Event 10046 trace

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2

983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29

983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270

sqlplus nolog demo_04_8sql 29

44

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582

POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF

bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec

RDTEX | Latch internals | RUOUG Seminar

12 ]2 )1[(

minus= +wait N

timeout

45

bull Most waits were for nothing ndash latch already was free

bull Latch utilization can not be high

bull Unnecessary spins provoke CPU thrashing

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682

92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE

sqlplus nolog demo_05sql 45

RDTEX | Latch internals | RUOUG Seminar

kslgetl(0x50006318 1)

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt kslges(0x50006318 ) -wait latch get

-gt skgslsgts(0x50006318 ) -spin latch get

-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT

-gt kskthbwt(0x0)

-gt kslwlmod() - set up Wait List

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt skgpwwait -sleep latch getsemop(11 17-10 1)

983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140

46

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782

RELIABLE LATCH WAITS

bull Hidden latch wait revolution that we missed Latch relies on waitposting

bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout

bull Exclusive latch spins 20000 cycles by default

bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock

bull Latches assigned to non-default class wait until timeout

bull Version 90 was transient and used mix of backoff and wait posting

This is why _LATCH_WAIT_POSTING parameter disappeared in9i

bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin

RDTEX | Latch internals | RUOUG Seminar

47

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882

IMAGINE THE TIME MICROSCOPE

| a a s | a

Reality One million zoomed time

1 us 1 sec

1 ms 17 min1 sec 115 days

Light speed (300000 kmsec) Sonic speed (300 msec)

CPU tick (2 GHz) 00005 sec

Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec

Max spin time for shared latch (~220005 ticks) ~10 sec

Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec

Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec

OS context switch time (10us-1ms) 10sec-17min

48

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982

LATCH WAITS UNDER TIME MICROSCOPE

Old 8i latch algorithmSpin for latch during 5 seconds

Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min

hellipSpin again for 5 seconds

Sleep again for 23 days (2 sec) hellip

New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)

If spin not succeeded go to sleep until post

According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)

| |

49

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082

_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES

bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and

_LATCH_CLASS_0 parameters

bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000

bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter

bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch

50

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182

TIMED LATCH WAIT POSTING

bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel

bull This can lead to instance hang ndash process will never be woken up

bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout

bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem

bull Generic AIX ports always use timed latch wait posting

bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux

51

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282

FINE GRAIN LATCH TUNING

bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting

bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7

bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo

microseconds using pollsys() (not semtimedop()) system call

ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times

Loop up to ldquoSpinsrdquo cycles

Yield CPU using yield() (or sched_yield())

ndash Sleep for ldquoSleepXrdquo usecs

ndash Then spin again hellip

52

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382

TRACES OF NONDEFAULT CLASS LATCH

983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152

991270

9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000

991270 E983158983141983150983156 10046 983156983154983137983139983141

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482

PERFORMANCE OF LATCH CLASSES

Row cache objects latch contention testcase on ODA (24 SMT)54

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582

IS THE _SPIN_COUNT REALLY STATIC

bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch

waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used

ldquocache buffers chainrdquo latch contention

bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass

(_LATCH_CLASS_X) By default ndash 20000

bull Shared latch spin can be tuned by _SPIN_COUNT value By

default ndash 2000 However this is not trivial

55

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682

SHARED LATCH BEHAVE LIKE ENQUEUE

bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80

bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests

bull This blocking achieved by special 0x40000000 bit in the latch

value This bit is a flag that some incompatible latch acquisitionis in progress

bull Queued processes will be woken up one by one on latch release

bull X mode latch gets effectively serialize the shared latch

bull See my blog for description of experiments

56

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782

TRACE OF SHARED LATCH WAIT

Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the

latch wait list and then spins again

983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141

983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156

983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983147983155983148983159983148983149983151983140()

983147983155983148983155983147983143983155(098316050009BAC )

983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140

983155983141983149983151983152(27)

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882

SHARED LATCH ACQUISITION

bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned

bull X mode shared latch get spins by default up to 4000 cycles

bull S mode does not spin at all (or spins in unknown way)

S mode get X mode get

Held in S mode Compatible 2_SPIN_COUNT

Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT

58

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982

LATCH RELEASE

bull Free the latch ndash kslfre(laddr)

bull Oracle process releases the latch nonatomically

bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process

bull This requires less bus invalidation and ensures propagation of latch

release to other local cachesbull Not fair policy Spinners on the local CPU board have the

preference

bull Then process posts the first process in the list of waiters

59

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082

LATCH CONTENTION

RDTEX | Latch internals | RUOUG Seminar60

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182

LATCH CONTENTION

bull Occurs when a latch is required by several processes at thesame time

bull Mostly due to bugs or abnormal application behavior

bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp

bull To diagnose the latch contention we need to investigate thelatch statistics [8]

61

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282

DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS

Latch requests arrival rate

Immediate gets efficiency

Latch sleeps ratio

Latch wait time per second

Latch spin efficiencymisses

getsspin∆

∆= _σ

misses

sleepsK

∆=

time

gets

∆=λ

gets

misses

∆= ρ

time

timewait W

∆=

_

Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics

62

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382

SAMPLED AVERAGES

By sampling of x$ksupr we can estimate

ndash Length of latch wait queue

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS

ndash Number of spinning processes in 10g

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS

Sampling of v$lathholder estimates latch utilization

Using DTrace we can measure

ndash Latch acquisition and holding times

ndash Actual spin count

w L

s N

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482

DERIVED LATCH STATISTICS

Latch utilization (PASTA)

Average holding time

Length of latch wait list

Recurrent sleeps ratio

Latch acquisition time

time

timeholdinglatchU

∆=asymp

__ ρ

Re_100

___

questsGet

TimeSnap MissGet Pct S

lowast==

λ

W L =

κ

κ σ 1minus+

)(1

W N T saq += minusλ

64

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 17: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1782

HOW ORACLEREQUESTS THE LATCH

RDTEX | Latch internals | RUOUG Seminar17

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1882

DTRACE REVEALS LATCH INTERFACE ROUTINES

Oracle calls the following functions to acquire the latch

ndash kslgetl(laddr wait why where) - get exclusive latch ndash kslg2c(l1l2trcwhy where) - get two excl child latches

ndash kslgetsl (laddrwaitwhywheremode) - get shared latch

bull 11g - ksl_get_shared_latch (hellip)

ndash kslg2cs(l1l2modetrcwhy where)) - get two shr child latches

ndash kslgpl(laddrcommentwhywhere) - get parent and all childs

ndash kslfre(laddr) - free the latch

Oracle give us possibility to do the same by oradebug call

RDTEX | Latch internals | RUOUG Seminar

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313501

983125983155983141 32983085983138983145983156 983152983151983154983156983155

64983138983145983156 O983154983137983139983148983141 983139983151983154983154983157983152983156983155 983156983144983141 983137983154983143983157983149983141983150983156983155 983151983142 983151983154983137983140983141983138983157983143 983139983137983148983148 983145983150 10204 983137983150983140 983137983138983151983158983141

18

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1982

ldquoWHERErdquo AND ldquoWHYrdquo ORACLE GETS THE LATCH

To request the latch Oracle kernel routines need

bull laddres ndash address of latch in SGAbull wait ndash flag for no-wait (0) or wait (1) latch acquisition

bull where ndash integer code for location from where the latch is acquired

Oracle lists possible latch ldquowhererdquo values in x$ksllwindxbull why - context of why the latch is acquiring at this ldquowhererdquo It

contains DBA for CBC latches (Tanel Poder) SGA chunk addressfor shared pool latch session address etchellip ldquoWhyrdquo meaning for

some ldquowhererdquo can be found in x$ksllwksllwlbl

bull mode ndash requesting state for shared lathes

ndash 8 SHARED mode 16 EXCLUSIVE mode

RDTEX | Latch internals | RUOUG Seminar

19

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2082

LIST OF LATCH ldquoWHERErdquo LOCATIONS

X$KSLLW ndash List of latch [W]here Part of v$latch_misses

ndash ksllwnam ndash Kernel function name(ldquokghupr1rdquo ldquoksqdelrdquo etc) ndash ksllwlbl ndash Description of ldquowhyrdquo context

Join this code on indx to

ndash x$ksuprlatksulawhr (v$latchholder)ndash ldquowhererdquo value that wasused by latch holder

ndash x$kslltkslltwhr (v$latch) ndash ldquowhererdquo that was used by lastsuccessful latch acquisition

ndash x$ksuprksllawer (v$process) ndash ldquowhererdquo that was used byprocess is being waiting for latch

ndash x$kslwscindx (v$latch_misses) ndash latch sleep statistics by

ldquowhererdquo

RDTEX | Latch internals | RUOUG Seminar

20

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2182

V$LATCH_MISSES

bull Based on X$KSLWSC - No[W]ait and [S]leep [C]ount stats byContext Documentation is rather obscure

bull The following counters are incremented on each latch sleep

sleep_count (ldquowhererdquo the latch was held)

wtr_slp_count (ldquowhererdquo the latch get was requested)

bull nwfail_count is incremented only for session which SID is equalto _LATCH_MISS_STAT_SID parameter

bull longhold_count is incremented for shared latches only when

someone held the latch at this ldquowhererdquo for the entire duration ofsleep

RDTEX | Latch internals | RUOUG Seminar

21

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2282

LATCH IS HOLDING BY PROCESS NOT SESSION

bull

RDTEX | Latch internals | RUOUG Seminar

983123983156983154983157983139983156 983147983155983157983152983154 983163

991270

983123983156983154983157983139983156 983147983155983148983148983137983163983147983155983148983148983156 98314798315598314898314898313798314898313798315698313114983133

983165

991270983165

983120983154983151983139983141983155983155 983142983145983160983141983140 983137983154983154983137983161

983158$983152983154983151983139983141983155983155 983085gt 983160$983147983155983157983152983154

983123983156983154983157983139983156 983147983155983148983148983156983163

991270

983165

L983145983155983156 983151983142 983137983148983148 983148983137983156983139983144983141983155

983158$983148983137983156983139983144 983085gt983160$983147983155983148983148983156

Each process has an array of references to the latches it is holding

22

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2382

KSLLA IS HIDDEN IN V$PROCESS

bull The process latching info is the kslla structure embedded in theprocess state object

bull Some kslla fields were externalized to SQL

bull kslla contains array of latches holding by process at each level andinformation about the state of any current attempt to acquire a

latch Not externalized directly

bull When you query v$latchholder x$ksuprlat Oracle scansv$processx$ksupr in order to find all the processes that are

holding the latchesbull Tanel Poder uses high frequency sampling of v$latchholder to

systematically troubleshoot latch contention

RDTEX | Latch internals | RUOUG Seminar

23

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2482

X$KSUPRKSLLA FIELDS INSTRUMENT THE LATCH GET

bull ksllalaq ndash address of latch acquiring Populated during immediateget (and spin before 11g)

bull ksllawat - latch being waited for This is v$processlatchwait

bull ksllawhy ndash ldquowhyrdquo for the latch being waited for

bull ksllawere ndash ldquowhererdquo for the latch being waited for

bull ksllalow ndash bit array of levels of currently holding latches

bull ksllaspn - latch this process is spinning on v$processlatchspinNot populated since 81

bull ksllaps - inter-process post statistics

RDTEX | Latch internals | RUOUG Seminar

24

Generic ldquolatch freerdquo and 27 latch specific ldquolatch free rdquo wait events areassociated with sleep only Spin is invisible to Oracle Wait Interface

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2582

THE LATCH STRUCTURE ndash KSLLT

RDTEX | Latch internals | RUOUG Seminar

struct ksllt

ltLatchgt

ldquowhererdquo and ldquowhyrdquo

Level latch class other attributes

Statistics

Latch wait list header

hellip

25

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2682

LATCH SIZE BY VERSION

nix 32bit nix 64bit Windows 32bit

734 92 - 120

806 104 - 104817 104 144 104

901 200 160

920 196 240 200

1010 256 208

1020 - 11203 100 160 104

RDTEX | Latch internals | RUOUG Seminar

x$ksmfsv ndash list of all fixed SGA variables983123E983116EC983124 DI983123983124I983118C983124 983147983155983149983142983155983155983145983162

F983122983119983117 983160$983147983155983149983142983155983158

983127HE983122E 983147983155983149983142983155983156983161983152 = 983147983155983148983148983156

Latch structure was bigger in 101 due to additional latch statistics

26

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2782

ORACLE LATCH IS NOT JUST A SINGLE MEMORY LOCATION

bull Before 11g Value of first latch byte (word for shared latches) wasused to determine latch state

0x00 ndash latch is free

0xFF ndash exclusive latch is busy Was 0x01 in Oracle 7

0x010x02hellip - shared latch holding by 12 hellip processes simultaneously

0x20000000 | pid - shared latch holding exclusively

bull In 11g first latch word shows the pid of the latch holder

0x00 ndash latch is free0x12 ndash Oracle process with pid 18 holds the exclusive latch

RDTEX | Latch internals | RUOUG Seminar

27

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2882

BUSY ORACLE 11G LATCH IN MEMORY

bull 112 Exclusive latch (32-bit platform)

bull 112 Shared latch EXCLUSIVE mode

RDTEX | Latch internals | RUOUG Seminar

SQLgt oradebug peek 200222A0 24

[200222A0 200222B8) = 00000016 00000001 000001D0 00000007

pid^ gets latch level

SQLgt oradebug peek 0xampladdr 24

[2000C814 2000C82C) = 2000001F 0000007C 0000007E 00000006 pid^ gets latch level

983139983151983150983156 983155983153983148983152983148983157983155 983150983151983148983151983143 9831409831419831499831519831350128

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2982

LATCH FIXED TABLES

bull Before Oracle 11g all the latches were externalized to SQL inv$latch (x$ksllt)

bull In 11g parent (fixed SGA) latches are externalized inv$latch_parent (x$kslltr_parent)

bull In 11g child (allocated on startup) latches are externalized in

v$latch_children (x$kslltr_children) bull Oracle 11g v$latch (x$kslltr) merges these data Beware select

from x$kslltr acquires all parent latches and their children

bull Only part of latch structure is visible in x$ksllthellip fixed tables ndash kslltwhr kslltwhy ndash ldquowhererdquo and ldquowhyrdquo latch was acquired last time

ndash kslltwgt hellip - statistics

ndash kslltlvl hellip - attributes

ndash kslltwtt ndash latch wait time

RDTEX | Latch internals | RUOUG Seminar

29

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3082

CUMULATIVE V$LATCH STATISTIC COUNTERS

RDTEX | Latch internals | RUOUG Seminar

Statistics x$ksllt When and how it increments

GETS kslltwgt ldquo++rdquo after wait mode latch get

MISSES kslltwff ldquo++rdquo after wait get if it was missed

SLEEPS kslltwsl ldquo+number_of_sleepsrdquo during get

SPIN_GETS ksllthst0 ldquo++rdquo if get was missed but not slept

WAIT_TIME kslltwtt ldquo+wait_timerdquo after latch get

IMMEDIATE_GETS kslltngt ldquo++rdquo after nowait mode latch get Is notincremented on miss

IMMEDIATE_MISSES kslltnfa ldquo++rdquo if nowait mode get was missed

30

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3182

MANY STATISTICS WERE LOST IN 102

In 102 Oracle made obsolete many useful latch statistics

Latch size was decreased and the following fields always contain 0now in v$latch

ndash v$latchwaiters_woken

ndash v$latchwaits_holding_latch

ndash v$latchsleep1hellipsleep10

ndash and many undocumented statistics

This was mentioned in 102 Reference

RDTEX | Latch internals | RUOUG Seminar

31

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3282

X$KSLLD - LATCH DESCRIPTORS

bull Flags and attributes describing each latch are stored in kslldt arrayof kslld structures

bull [K]ernel [S]ervice [L]atch [L]ock [D]escriptor

bull Some but not all of kslld fields are externalized via x$kslld andv$latchname

bull Located in the PGA

RDTEX | Latch internals | RUOUG Seminar

32

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382

LATCH ATTRIBUTES

bull Each latch have at least the following attributes in kslldt

ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared

ndash PAR Is the latch Solitary or Parent for the family of child latches

ndash G2C Can two child latches be simultaneously requested in wait mode

using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92

ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above

ndash Level 0-14 Used to prevent latch deadlocks

ndash Class 0-7 Spin and wait class assigned to the latch 92 and above

RDTEX | Latch internals | RUOUG Seminar

33

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482

LATCH ATTRIBUTES BY ORACLE VERSION

RDTEX | Latch internals | RUOUG Seminar

11203 553 154 72 - 6 93

Oracle Number of latches PAR G2C LNG UFS SHARED

7340 53 14 2 3 - -

8063 80 21 7 3 - 38174 152 48 19 4 - 9

9208 242 79 37 - - 19

10202 385 114 55 - 4 47

10203 388 117 58 - 4 48

10204 394 117 59 - 4 50

11106 496 145 67 - 6 81

11107 502 145 67 - 6 83

11201 535 149 70 - 6 86

11202 551 153 72 - 6 91

34

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582

LATCH LEVELS

bull Every latch has a level in the range 0-14 to avoid deadlocks

bull Process can request a latch X in wait mode after obtaining latch Y if and

only if level X gt level Y

bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y

bull If these rules are broken the Oracle process raises

ORA-600[504] - trying to obtain another latch at incompatible level Note 281041

ORA-600[525] - trying to obtain another latch at the same level Note 1388881

ORA-600[526] - latch child order violation Note 1388891

RDTEX | Latch internals | RUOUG Seminar

35

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682

PARENT AND CHILD LATCHES

bull Latch can be a parent child or solitary latch

bull Both parent and child latches share the same latch name

bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss

bull Child latches are allocated during instance startup

bull The parent latch can be gotten independently

bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s

RDTEX | Latch internals | RUOUG Seminar

(latch info) wait_event=0 bits=10

holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl

Context saved from call 0

state=busy

36

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782

LATCH TREES

ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches

Direct SGA access program output for 9206 instance with too smallshared pool

RDTEX | Latch internals | RUOUG Seminar

ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1

waiter ospid 129 sid 72 pid 45

holding a154b7120 library cache (157) level=5 child=17 kglupc child

waiter ospid 18255 sid 65 pid 930

waiter ospid 6690 sid 554 pid 1654

waiter ospid 4685 sid 879 pid 1034

hellip

waiter ospid 29749 sid 180 pid 155

holding a154b7db8 library cache (157) level=5 child=4 kglupc child

waiter ospid 13104 sid 281 pid 220

waiter ospid 24089 sid 565 pid 636

waiter ospid 25002 sid 621 pid 1481

waiter ospid 16930 sid 1046 pid 783hellip

37

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882

LATCH CLASSES

bull Classes with different spin and wait policies

bull Up to 8 classes By default all the latches belong to class 0

bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2

bull Latch can be assigned to class by its number For example

ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000

ndash _LATCH_CLASSES=46

bull Will be discussed later

RDTEX | Latch internals | RUOUG Seminar

SQLgt select from x$ksllclass

INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4

---- ------ ----- -------- ------ ------ ------ ------ ------

0 20000 0 1 8000 8000 8000 8000 8000

hellip

6 100 12 1000 2000 3000 4000 4000 4000

38

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982

LATCH ACQUISITION IN NO-WAIT MODE

bull kslgetl(laddr0whywhere) ndash Exclusive latch get

bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep

ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches

ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches

bull IMMEDIATE_ statistics are updated

bull Used when many candidate latches are available to bypass the

latch level rule or as part of CBC chain examination

RDTEX | Latch internals | RUOUG Seminar

sqlplus nolog demo_03sql39

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082

LATCH WAITS

RDTEX | Latch internals | RUOUG Seminar 40

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182

WAITING FOR THE LATCH

RDTEX | Latch internals | RUOUG Seminar

983123 G A

C983120983125 1 C983120983125 2

983120983154983151983139983141983155983155 A

983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)

983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144

983116983137983156983139983144

983120983154983151983139983141983155983155 B

41

RDTEX | L h i l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282

LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION

bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries

again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of

times

bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep

bull A sleep corresponds to a wait for the latch free wait event

bull Initially a process sleeps for 001 s This time is doubled in every

subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip

RDTEX | Latch internals | RUOUG Seminar

Still did not found any versionwhich used this

May be it was lt= Oracle 6

42

RDTEX | L t h i t l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382

LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81

Latch gets with wait (kslgetl(laddress1hellip))

ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times

ndash Sleep on latch free event with exponential backoff

ndash Repeat

RDTEX | Latch internals | RUOUG Seminar

43

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482

8I LATCH SPIN CODE FLOW USING DTRACE

RDTEX | Latch internals | RUOUG Seminar

bull hellip Event 10046 trace

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2

983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29

983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270

sqlplus nolog demo_04_8sql 29

44

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582

POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF

bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec

RDTEX | Latch internals | RUOUG Seminar

12 ]2 )1[(

minus= +wait N

timeout

45

bull Most waits were for nothing ndash latch already was free

bull Latch utilization can not be high

bull Unnecessary spins provoke CPU thrashing

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682

92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE

sqlplus nolog demo_05sql 45

RDTEX | Latch internals | RUOUG Seminar

kslgetl(0x50006318 1)

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt kslges(0x50006318 ) -wait latch get

-gt skgslsgts(0x50006318 ) -spin latch get

-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT

-gt kskthbwt(0x0)

-gt kslwlmod() - set up Wait List

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt skgpwwait -sleep latch getsemop(11 17-10 1)

983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140

46

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782

RELIABLE LATCH WAITS

bull Hidden latch wait revolution that we missed Latch relies on waitposting

bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout

bull Exclusive latch spins 20000 cycles by default

bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock

bull Latches assigned to non-default class wait until timeout

bull Version 90 was transient and used mix of backoff and wait posting

This is why _LATCH_WAIT_POSTING parameter disappeared in9i

bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin

RDTEX | Latch internals | RUOUG Seminar

47

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882

IMAGINE THE TIME MICROSCOPE

| a a s | a

Reality One million zoomed time

1 us 1 sec

1 ms 17 min1 sec 115 days

Light speed (300000 kmsec) Sonic speed (300 msec)

CPU tick (2 GHz) 00005 sec

Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec

Max spin time for shared latch (~220005 ticks) ~10 sec

Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec

Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec

OS context switch time (10us-1ms) 10sec-17min

48

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982

LATCH WAITS UNDER TIME MICROSCOPE

Old 8i latch algorithmSpin for latch during 5 seconds

Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min

hellipSpin again for 5 seconds

Sleep again for 23 days (2 sec) hellip

New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)

If spin not succeeded go to sleep until post

According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)

| |

49

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082

_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES

bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and

_LATCH_CLASS_0 parameters

bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000

bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter

bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch

50

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182

TIMED LATCH WAIT POSTING

bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel

bull This can lead to instance hang ndash process will never be woken up

bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout

bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem

bull Generic AIX ports always use timed latch wait posting

bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux

51

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282

FINE GRAIN LATCH TUNING

bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting

bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7

bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo

microseconds using pollsys() (not semtimedop()) system call

ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times

Loop up to ldquoSpinsrdquo cycles

Yield CPU using yield() (or sched_yield())

ndash Sleep for ldquoSleepXrdquo usecs

ndash Then spin again hellip

52

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382

TRACES OF NONDEFAULT CLASS LATCH

983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152

991270

9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000

991270 E983158983141983150983156 10046 983156983154983137983139983141

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482

PERFORMANCE OF LATCH CLASSES

Row cache objects latch contention testcase on ODA (24 SMT)54

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582

IS THE _SPIN_COUNT REALLY STATIC

bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch

waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used

ldquocache buffers chainrdquo latch contention

bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass

(_LATCH_CLASS_X) By default ndash 20000

bull Shared latch spin can be tuned by _SPIN_COUNT value By

default ndash 2000 However this is not trivial

55

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682

SHARED LATCH BEHAVE LIKE ENQUEUE

bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80

bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests

bull This blocking achieved by special 0x40000000 bit in the latch

value This bit is a flag that some incompatible latch acquisitionis in progress

bull Queued processes will be woken up one by one on latch release

bull X mode latch gets effectively serialize the shared latch

bull See my blog for description of experiments

56

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782

TRACE OF SHARED LATCH WAIT

Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the

latch wait list and then spins again

983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141

983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156

983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983147983155983148983159983148983149983151983140()

983147983155983148983155983147983143983155(098316050009BAC )

983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140

983155983141983149983151983152(27)

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882

SHARED LATCH ACQUISITION

bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned

bull X mode shared latch get spins by default up to 4000 cycles

bull S mode does not spin at all (or spins in unknown way)

S mode get X mode get

Held in S mode Compatible 2_SPIN_COUNT

Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT

58

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982

LATCH RELEASE

bull Free the latch ndash kslfre(laddr)

bull Oracle process releases the latch nonatomically

bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process

bull This requires less bus invalidation and ensures propagation of latch

release to other local cachesbull Not fair policy Spinners on the local CPU board have the

preference

bull Then process posts the first process in the list of waiters

59

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082

LATCH CONTENTION

RDTEX | Latch internals | RUOUG Seminar60

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182

LATCH CONTENTION

bull Occurs when a latch is required by several processes at thesame time

bull Mostly due to bugs or abnormal application behavior

bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp

bull To diagnose the latch contention we need to investigate thelatch statistics [8]

61

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282

DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS

Latch requests arrival rate

Immediate gets efficiency

Latch sleeps ratio

Latch wait time per second

Latch spin efficiencymisses

getsspin∆

∆= _σ

misses

sleepsK

∆=

time

gets

∆=λ

gets

misses

∆= ρ

time

timewait W

∆=

_

Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics

62

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382

SAMPLED AVERAGES

By sampling of x$ksupr we can estimate

ndash Length of latch wait queue

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS

ndash Number of spinning processes in 10g

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS

Sampling of v$lathholder estimates latch utilization

Using DTrace we can measure

ndash Latch acquisition and holding times

ndash Actual spin count

w L

s N

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482

DERIVED LATCH STATISTICS

Latch utilization (PASTA)

Average holding time

Length of latch wait list

Recurrent sleeps ratio

Latch acquisition time

time

timeholdinglatchU

∆=asymp

__ ρ

Re_100

___

questsGet

TimeSnap MissGet Pct S

lowast==

λ

W L =

κ

κ σ 1minus+

)(1

W N T saq += minusλ

64

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 18: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1882

DTRACE REVEALS LATCH INTERFACE ROUTINES

Oracle calls the following functions to acquire the latch

ndash kslgetl(laddr wait why where) - get exclusive latch ndash kslg2c(l1l2trcwhy where) - get two excl child latches

ndash kslgetsl (laddrwaitwhywheremode) - get shared latch

bull 11g - ksl_get_shared_latch (hellip)

ndash kslg2cs(l1l2modetrcwhy where)) - get two shr child latches

ndash kslgpl(laddrcommentwhywhere) - get parent and all childs

ndash kslfre(laddr) - free the latch

Oracle give us possibility to do the same by oradebug call

RDTEX | Latch internals | RUOUG Seminar

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313501

983125983155983141 32983085983138983145983156 983152983151983154983156983155

64983138983145983156 O983154983137983139983148983141 983139983151983154983154983157983152983156983155 983156983144983141 983137983154983143983157983149983141983150983156983155 983151983142 983151983154983137983140983141983138983157983143 983139983137983148983148 983145983150 10204 983137983150983140 983137983138983151983158983141

18

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1982

ldquoWHERErdquo AND ldquoWHYrdquo ORACLE GETS THE LATCH

To request the latch Oracle kernel routines need

bull laddres ndash address of latch in SGAbull wait ndash flag for no-wait (0) or wait (1) latch acquisition

bull where ndash integer code for location from where the latch is acquired

Oracle lists possible latch ldquowhererdquo values in x$ksllwindxbull why - context of why the latch is acquiring at this ldquowhererdquo It

contains DBA for CBC latches (Tanel Poder) SGA chunk addressfor shared pool latch session address etchellip ldquoWhyrdquo meaning for

some ldquowhererdquo can be found in x$ksllwksllwlbl

bull mode ndash requesting state for shared lathes

ndash 8 SHARED mode 16 EXCLUSIVE mode

RDTEX | Latch internals | RUOUG Seminar

19

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2082

LIST OF LATCH ldquoWHERErdquo LOCATIONS

X$KSLLW ndash List of latch [W]here Part of v$latch_misses

ndash ksllwnam ndash Kernel function name(ldquokghupr1rdquo ldquoksqdelrdquo etc) ndash ksllwlbl ndash Description of ldquowhyrdquo context

Join this code on indx to

ndash x$ksuprlatksulawhr (v$latchholder)ndash ldquowhererdquo value that wasused by latch holder

ndash x$kslltkslltwhr (v$latch) ndash ldquowhererdquo that was used by lastsuccessful latch acquisition

ndash x$ksuprksllawer (v$process) ndash ldquowhererdquo that was used byprocess is being waiting for latch

ndash x$kslwscindx (v$latch_misses) ndash latch sleep statistics by

ldquowhererdquo

RDTEX | Latch internals | RUOUG Seminar

20

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2182

V$LATCH_MISSES

bull Based on X$KSLWSC - No[W]ait and [S]leep [C]ount stats byContext Documentation is rather obscure

bull The following counters are incremented on each latch sleep

sleep_count (ldquowhererdquo the latch was held)

wtr_slp_count (ldquowhererdquo the latch get was requested)

bull nwfail_count is incremented only for session which SID is equalto _LATCH_MISS_STAT_SID parameter

bull longhold_count is incremented for shared latches only when

someone held the latch at this ldquowhererdquo for the entire duration ofsleep

RDTEX | Latch internals | RUOUG Seminar

21

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2282

LATCH IS HOLDING BY PROCESS NOT SESSION

bull

RDTEX | Latch internals | RUOUG Seminar

983123983156983154983157983139983156 983147983155983157983152983154 983163

991270

983123983156983154983157983139983156 983147983155983148983148983137983163983147983155983148983148983156 98314798315598314898314898313798314898313798315698313114983133

983165

991270983165

983120983154983151983139983141983155983155 983142983145983160983141983140 983137983154983154983137983161

983158$983152983154983151983139983141983155983155 983085gt 983160$983147983155983157983152983154

983123983156983154983157983139983156 983147983155983148983148983156983163

991270

983165

L983145983155983156 983151983142 983137983148983148 983148983137983156983139983144983141983155

983158$983148983137983156983139983144 983085gt983160$983147983155983148983148983156

Each process has an array of references to the latches it is holding

22

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2382

KSLLA IS HIDDEN IN V$PROCESS

bull The process latching info is the kslla structure embedded in theprocess state object

bull Some kslla fields were externalized to SQL

bull kslla contains array of latches holding by process at each level andinformation about the state of any current attempt to acquire a

latch Not externalized directly

bull When you query v$latchholder x$ksuprlat Oracle scansv$processx$ksupr in order to find all the processes that are

holding the latchesbull Tanel Poder uses high frequency sampling of v$latchholder to

systematically troubleshoot latch contention

RDTEX | Latch internals | RUOUG Seminar

23

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2482

X$KSUPRKSLLA FIELDS INSTRUMENT THE LATCH GET

bull ksllalaq ndash address of latch acquiring Populated during immediateget (and spin before 11g)

bull ksllawat - latch being waited for This is v$processlatchwait

bull ksllawhy ndash ldquowhyrdquo for the latch being waited for

bull ksllawere ndash ldquowhererdquo for the latch being waited for

bull ksllalow ndash bit array of levels of currently holding latches

bull ksllaspn - latch this process is spinning on v$processlatchspinNot populated since 81

bull ksllaps - inter-process post statistics

RDTEX | Latch internals | RUOUG Seminar

24

Generic ldquolatch freerdquo and 27 latch specific ldquolatch free rdquo wait events areassociated with sleep only Spin is invisible to Oracle Wait Interface

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2582

THE LATCH STRUCTURE ndash KSLLT

RDTEX | Latch internals | RUOUG Seminar

struct ksllt

ltLatchgt

ldquowhererdquo and ldquowhyrdquo

Level latch class other attributes

Statistics

Latch wait list header

hellip

25

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2682

LATCH SIZE BY VERSION

nix 32bit nix 64bit Windows 32bit

734 92 - 120

806 104 - 104817 104 144 104

901 200 160

920 196 240 200

1010 256 208

1020 - 11203 100 160 104

RDTEX | Latch internals | RUOUG Seminar

x$ksmfsv ndash list of all fixed SGA variables983123E983116EC983124 DI983123983124I983118C983124 983147983155983149983142983155983155983145983162

F983122983119983117 983160$983147983155983149983142983155983158

983127HE983122E 983147983155983149983142983155983156983161983152 = 983147983155983148983148983156

Latch structure was bigger in 101 due to additional latch statistics

26

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2782

ORACLE LATCH IS NOT JUST A SINGLE MEMORY LOCATION

bull Before 11g Value of first latch byte (word for shared latches) wasused to determine latch state

0x00 ndash latch is free

0xFF ndash exclusive latch is busy Was 0x01 in Oracle 7

0x010x02hellip - shared latch holding by 12 hellip processes simultaneously

0x20000000 | pid - shared latch holding exclusively

bull In 11g first latch word shows the pid of the latch holder

0x00 ndash latch is free0x12 ndash Oracle process with pid 18 holds the exclusive latch

RDTEX | Latch internals | RUOUG Seminar

27

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2882

BUSY ORACLE 11G LATCH IN MEMORY

bull 112 Exclusive latch (32-bit platform)

bull 112 Shared latch EXCLUSIVE mode

RDTEX | Latch internals | RUOUG Seminar

SQLgt oradebug peek 200222A0 24

[200222A0 200222B8) = 00000016 00000001 000001D0 00000007

pid^ gets latch level

SQLgt oradebug peek 0xampladdr 24

[2000C814 2000C82C) = 2000001F 0000007C 0000007E 00000006 pid^ gets latch level

983139983151983150983156 983155983153983148983152983148983157983155 983150983151983148983151983143 9831409831419831499831519831350128

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2982

LATCH FIXED TABLES

bull Before Oracle 11g all the latches were externalized to SQL inv$latch (x$ksllt)

bull In 11g parent (fixed SGA) latches are externalized inv$latch_parent (x$kslltr_parent)

bull In 11g child (allocated on startup) latches are externalized in

v$latch_children (x$kslltr_children) bull Oracle 11g v$latch (x$kslltr) merges these data Beware select

from x$kslltr acquires all parent latches and their children

bull Only part of latch structure is visible in x$ksllthellip fixed tables ndash kslltwhr kslltwhy ndash ldquowhererdquo and ldquowhyrdquo latch was acquired last time

ndash kslltwgt hellip - statistics

ndash kslltlvl hellip - attributes

ndash kslltwtt ndash latch wait time

RDTEX | Latch internals | RUOUG Seminar

29

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3082

CUMULATIVE V$LATCH STATISTIC COUNTERS

RDTEX | Latch internals | RUOUG Seminar

Statistics x$ksllt When and how it increments

GETS kslltwgt ldquo++rdquo after wait mode latch get

MISSES kslltwff ldquo++rdquo after wait get if it was missed

SLEEPS kslltwsl ldquo+number_of_sleepsrdquo during get

SPIN_GETS ksllthst0 ldquo++rdquo if get was missed but not slept

WAIT_TIME kslltwtt ldquo+wait_timerdquo after latch get

IMMEDIATE_GETS kslltngt ldquo++rdquo after nowait mode latch get Is notincremented on miss

IMMEDIATE_MISSES kslltnfa ldquo++rdquo if nowait mode get was missed

30

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3182

MANY STATISTICS WERE LOST IN 102

In 102 Oracle made obsolete many useful latch statistics

Latch size was decreased and the following fields always contain 0now in v$latch

ndash v$latchwaiters_woken

ndash v$latchwaits_holding_latch

ndash v$latchsleep1hellipsleep10

ndash and many undocumented statistics

This was mentioned in 102 Reference

RDTEX | Latch internals | RUOUG Seminar

31

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3282

X$KSLLD - LATCH DESCRIPTORS

bull Flags and attributes describing each latch are stored in kslldt arrayof kslld structures

bull [K]ernel [S]ervice [L]atch [L]ock [D]escriptor

bull Some but not all of kslld fields are externalized via x$kslld andv$latchname

bull Located in the PGA

RDTEX | Latch internals | RUOUG Seminar

32

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382

LATCH ATTRIBUTES

bull Each latch have at least the following attributes in kslldt

ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared

ndash PAR Is the latch Solitary or Parent for the family of child latches

ndash G2C Can two child latches be simultaneously requested in wait mode

using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92

ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above

ndash Level 0-14 Used to prevent latch deadlocks

ndash Class 0-7 Spin and wait class assigned to the latch 92 and above

RDTEX | Latch internals | RUOUG Seminar

33

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482

LATCH ATTRIBUTES BY ORACLE VERSION

RDTEX | Latch internals | RUOUG Seminar

11203 553 154 72 - 6 93

Oracle Number of latches PAR G2C LNG UFS SHARED

7340 53 14 2 3 - -

8063 80 21 7 3 - 38174 152 48 19 4 - 9

9208 242 79 37 - - 19

10202 385 114 55 - 4 47

10203 388 117 58 - 4 48

10204 394 117 59 - 4 50

11106 496 145 67 - 6 81

11107 502 145 67 - 6 83

11201 535 149 70 - 6 86

11202 551 153 72 - 6 91

34

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582

LATCH LEVELS

bull Every latch has a level in the range 0-14 to avoid deadlocks

bull Process can request a latch X in wait mode after obtaining latch Y if and

only if level X gt level Y

bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y

bull If these rules are broken the Oracle process raises

ORA-600[504] - trying to obtain another latch at incompatible level Note 281041

ORA-600[525] - trying to obtain another latch at the same level Note 1388881

ORA-600[526] - latch child order violation Note 1388891

RDTEX | Latch internals | RUOUG Seminar

35

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682

PARENT AND CHILD LATCHES

bull Latch can be a parent child or solitary latch

bull Both parent and child latches share the same latch name

bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss

bull Child latches are allocated during instance startup

bull The parent latch can be gotten independently

bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s

RDTEX | Latch internals | RUOUG Seminar

(latch info) wait_event=0 bits=10

holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl

Context saved from call 0

state=busy

36

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782

LATCH TREES

ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches

Direct SGA access program output for 9206 instance with too smallshared pool

RDTEX | Latch internals | RUOUG Seminar

ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1

waiter ospid 129 sid 72 pid 45

holding a154b7120 library cache (157) level=5 child=17 kglupc child

waiter ospid 18255 sid 65 pid 930

waiter ospid 6690 sid 554 pid 1654

waiter ospid 4685 sid 879 pid 1034

hellip

waiter ospid 29749 sid 180 pid 155

holding a154b7db8 library cache (157) level=5 child=4 kglupc child

waiter ospid 13104 sid 281 pid 220

waiter ospid 24089 sid 565 pid 636

waiter ospid 25002 sid 621 pid 1481

waiter ospid 16930 sid 1046 pid 783hellip

37

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882

LATCH CLASSES

bull Classes with different spin and wait policies

bull Up to 8 classes By default all the latches belong to class 0

bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2

bull Latch can be assigned to class by its number For example

ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000

ndash _LATCH_CLASSES=46

bull Will be discussed later

RDTEX | Latch internals | RUOUG Seminar

SQLgt select from x$ksllclass

INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4

---- ------ ----- -------- ------ ------ ------ ------ ------

0 20000 0 1 8000 8000 8000 8000 8000

hellip

6 100 12 1000 2000 3000 4000 4000 4000

38

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982

LATCH ACQUISITION IN NO-WAIT MODE

bull kslgetl(laddr0whywhere) ndash Exclusive latch get

bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep

ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches

ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches

bull IMMEDIATE_ statistics are updated

bull Used when many candidate latches are available to bypass the

latch level rule or as part of CBC chain examination

RDTEX | Latch internals | RUOUG Seminar

sqlplus nolog demo_03sql39

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082

LATCH WAITS

RDTEX | Latch internals | RUOUG Seminar 40

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182

WAITING FOR THE LATCH

RDTEX | Latch internals | RUOUG Seminar

983123 G A

C983120983125 1 C983120983125 2

983120983154983151983139983141983155983155 A

983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)

983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144

983116983137983156983139983144

983120983154983151983139983141983155983155 B

41

RDTEX | L h i l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282

LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION

bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries

again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of

times

bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep

bull A sleep corresponds to a wait for the latch free wait event

bull Initially a process sleeps for 001 s This time is doubled in every

subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip

RDTEX | Latch internals | RUOUG Seminar

Still did not found any versionwhich used this

May be it was lt= Oracle 6

42

RDTEX | L t h i t l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382

LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81

Latch gets with wait (kslgetl(laddress1hellip))

ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times

ndash Sleep on latch free event with exponential backoff

ndash Repeat

RDTEX | Latch internals | RUOUG Seminar

43

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482

8I LATCH SPIN CODE FLOW USING DTRACE

RDTEX | Latch internals | RUOUG Seminar

bull hellip Event 10046 trace

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2

983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29

983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270

sqlplus nolog demo_04_8sql 29

44

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582

POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF

bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec

RDTEX | Latch internals | RUOUG Seminar

12 ]2 )1[(

minus= +wait N

timeout

45

bull Most waits were for nothing ndash latch already was free

bull Latch utilization can not be high

bull Unnecessary spins provoke CPU thrashing

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682

92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE

sqlplus nolog demo_05sql 45

RDTEX | Latch internals | RUOUG Seminar

kslgetl(0x50006318 1)

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt kslges(0x50006318 ) -wait latch get

-gt skgslsgts(0x50006318 ) -spin latch get

-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT

-gt kskthbwt(0x0)

-gt kslwlmod() - set up Wait List

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt skgpwwait -sleep latch getsemop(11 17-10 1)

983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140

46

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782

RELIABLE LATCH WAITS

bull Hidden latch wait revolution that we missed Latch relies on waitposting

bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout

bull Exclusive latch spins 20000 cycles by default

bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock

bull Latches assigned to non-default class wait until timeout

bull Version 90 was transient and used mix of backoff and wait posting

This is why _LATCH_WAIT_POSTING parameter disappeared in9i

bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin

RDTEX | Latch internals | RUOUG Seminar

47

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882

IMAGINE THE TIME MICROSCOPE

| a a s | a

Reality One million zoomed time

1 us 1 sec

1 ms 17 min1 sec 115 days

Light speed (300000 kmsec) Sonic speed (300 msec)

CPU tick (2 GHz) 00005 sec

Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec

Max spin time for shared latch (~220005 ticks) ~10 sec

Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec

Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec

OS context switch time (10us-1ms) 10sec-17min

48

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982

LATCH WAITS UNDER TIME MICROSCOPE

Old 8i latch algorithmSpin for latch during 5 seconds

Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min

hellipSpin again for 5 seconds

Sleep again for 23 days (2 sec) hellip

New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)

If spin not succeeded go to sleep until post

According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)

| |

49

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082

_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES

bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and

_LATCH_CLASS_0 parameters

bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000

bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter

bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch

50

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182

TIMED LATCH WAIT POSTING

bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel

bull This can lead to instance hang ndash process will never be woken up

bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout

bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem

bull Generic AIX ports always use timed latch wait posting

bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux

51

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282

FINE GRAIN LATCH TUNING

bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting

bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7

bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo

microseconds using pollsys() (not semtimedop()) system call

ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times

Loop up to ldquoSpinsrdquo cycles

Yield CPU using yield() (or sched_yield())

ndash Sleep for ldquoSleepXrdquo usecs

ndash Then spin again hellip

52

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382

TRACES OF NONDEFAULT CLASS LATCH

983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152

991270

9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000

991270 E983158983141983150983156 10046 983156983154983137983139983141

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482

PERFORMANCE OF LATCH CLASSES

Row cache objects latch contention testcase on ODA (24 SMT)54

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582

IS THE _SPIN_COUNT REALLY STATIC

bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch

waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used

ldquocache buffers chainrdquo latch contention

bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass

(_LATCH_CLASS_X) By default ndash 20000

bull Shared latch spin can be tuned by _SPIN_COUNT value By

default ndash 2000 However this is not trivial

55

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682

SHARED LATCH BEHAVE LIKE ENQUEUE

bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80

bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests

bull This blocking achieved by special 0x40000000 bit in the latch

value This bit is a flag that some incompatible latch acquisitionis in progress

bull Queued processes will be woken up one by one on latch release

bull X mode latch gets effectively serialize the shared latch

bull See my blog for description of experiments

56

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782

TRACE OF SHARED LATCH WAIT

Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the

latch wait list and then spins again

983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141

983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156

983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983147983155983148983159983148983149983151983140()

983147983155983148983155983147983143983155(098316050009BAC )

983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140

983155983141983149983151983152(27)

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882

SHARED LATCH ACQUISITION

bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned

bull X mode shared latch get spins by default up to 4000 cycles

bull S mode does not spin at all (or spins in unknown way)

S mode get X mode get

Held in S mode Compatible 2_SPIN_COUNT

Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT

58

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982

LATCH RELEASE

bull Free the latch ndash kslfre(laddr)

bull Oracle process releases the latch nonatomically

bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process

bull This requires less bus invalidation and ensures propagation of latch

release to other local cachesbull Not fair policy Spinners on the local CPU board have the

preference

bull Then process posts the first process in the list of waiters

59

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082

LATCH CONTENTION

RDTEX | Latch internals | RUOUG Seminar60

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182

LATCH CONTENTION

bull Occurs when a latch is required by several processes at thesame time

bull Mostly due to bugs or abnormal application behavior

bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp

bull To diagnose the latch contention we need to investigate thelatch statistics [8]

61

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282

DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS

Latch requests arrival rate

Immediate gets efficiency

Latch sleeps ratio

Latch wait time per second

Latch spin efficiencymisses

getsspin∆

∆= _σ

misses

sleepsK

∆=

time

gets

∆=λ

gets

misses

∆= ρ

time

timewait W

∆=

_

Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics

62

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382

SAMPLED AVERAGES

By sampling of x$ksupr we can estimate

ndash Length of latch wait queue

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS

ndash Number of spinning processes in 10g

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS

Sampling of v$lathholder estimates latch utilization

Using DTrace we can measure

ndash Latch acquisition and holding times

ndash Actual spin count

w L

s N

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482

DERIVED LATCH STATISTICS

Latch utilization (PASTA)

Average holding time

Length of latch wait list

Recurrent sleeps ratio

Latch acquisition time

time

timeholdinglatchU

∆=asymp

__ ρ

Re_100

___

questsGet

TimeSnap MissGet Pct S

lowast==

λ

W L =

κ

κ σ 1minus+

)(1

W N T saq += minusλ

64

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 19: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 1982

ldquoWHERErdquo AND ldquoWHYrdquo ORACLE GETS THE LATCH

To request the latch Oracle kernel routines need

bull laddres ndash address of latch in SGAbull wait ndash flag for no-wait (0) or wait (1) latch acquisition

bull where ndash integer code for location from where the latch is acquired

Oracle lists possible latch ldquowhererdquo values in x$ksllwindxbull why - context of why the latch is acquiring at this ldquowhererdquo It

contains DBA for CBC latches (Tanel Poder) SGA chunk addressfor shared pool latch session address etchellip ldquoWhyrdquo meaning for

some ldquowhererdquo can be found in x$ksllwksllwlbl

bull mode ndash requesting state for shared lathes

ndash 8 SHARED mode 16 EXCLUSIVE mode

RDTEX | Latch internals | RUOUG Seminar

19

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2082

LIST OF LATCH ldquoWHERErdquo LOCATIONS

X$KSLLW ndash List of latch [W]here Part of v$latch_misses

ndash ksllwnam ndash Kernel function name(ldquokghupr1rdquo ldquoksqdelrdquo etc) ndash ksllwlbl ndash Description of ldquowhyrdquo context

Join this code on indx to

ndash x$ksuprlatksulawhr (v$latchholder)ndash ldquowhererdquo value that wasused by latch holder

ndash x$kslltkslltwhr (v$latch) ndash ldquowhererdquo that was used by lastsuccessful latch acquisition

ndash x$ksuprksllawer (v$process) ndash ldquowhererdquo that was used byprocess is being waiting for latch

ndash x$kslwscindx (v$latch_misses) ndash latch sleep statistics by

ldquowhererdquo

RDTEX | Latch internals | RUOUG Seminar

20

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2182

V$LATCH_MISSES

bull Based on X$KSLWSC - No[W]ait and [S]leep [C]ount stats byContext Documentation is rather obscure

bull The following counters are incremented on each latch sleep

sleep_count (ldquowhererdquo the latch was held)

wtr_slp_count (ldquowhererdquo the latch get was requested)

bull nwfail_count is incremented only for session which SID is equalto _LATCH_MISS_STAT_SID parameter

bull longhold_count is incremented for shared latches only when

someone held the latch at this ldquowhererdquo for the entire duration ofsleep

RDTEX | Latch internals | RUOUG Seminar

21

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2282

LATCH IS HOLDING BY PROCESS NOT SESSION

bull

RDTEX | Latch internals | RUOUG Seminar

983123983156983154983157983139983156 983147983155983157983152983154 983163

991270

983123983156983154983157983139983156 983147983155983148983148983137983163983147983155983148983148983156 98314798315598314898314898313798314898313798315698313114983133

983165

991270983165

983120983154983151983139983141983155983155 983142983145983160983141983140 983137983154983154983137983161

983158$983152983154983151983139983141983155983155 983085gt 983160$983147983155983157983152983154

983123983156983154983157983139983156 983147983155983148983148983156983163

991270

983165

L983145983155983156 983151983142 983137983148983148 983148983137983156983139983144983141983155

983158$983148983137983156983139983144 983085gt983160$983147983155983148983148983156

Each process has an array of references to the latches it is holding

22

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2382

KSLLA IS HIDDEN IN V$PROCESS

bull The process latching info is the kslla structure embedded in theprocess state object

bull Some kslla fields were externalized to SQL

bull kslla contains array of latches holding by process at each level andinformation about the state of any current attempt to acquire a

latch Not externalized directly

bull When you query v$latchholder x$ksuprlat Oracle scansv$processx$ksupr in order to find all the processes that are

holding the latchesbull Tanel Poder uses high frequency sampling of v$latchholder to

systematically troubleshoot latch contention

RDTEX | Latch internals | RUOUG Seminar

23

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2482

X$KSUPRKSLLA FIELDS INSTRUMENT THE LATCH GET

bull ksllalaq ndash address of latch acquiring Populated during immediateget (and spin before 11g)

bull ksllawat - latch being waited for This is v$processlatchwait

bull ksllawhy ndash ldquowhyrdquo for the latch being waited for

bull ksllawere ndash ldquowhererdquo for the latch being waited for

bull ksllalow ndash bit array of levels of currently holding latches

bull ksllaspn - latch this process is spinning on v$processlatchspinNot populated since 81

bull ksllaps - inter-process post statistics

RDTEX | Latch internals | RUOUG Seminar

24

Generic ldquolatch freerdquo and 27 latch specific ldquolatch free rdquo wait events areassociated with sleep only Spin is invisible to Oracle Wait Interface

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2582

THE LATCH STRUCTURE ndash KSLLT

RDTEX | Latch internals | RUOUG Seminar

struct ksllt

ltLatchgt

ldquowhererdquo and ldquowhyrdquo

Level latch class other attributes

Statistics

Latch wait list header

hellip

25

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2682

LATCH SIZE BY VERSION

nix 32bit nix 64bit Windows 32bit

734 92 - 120

806 104 - 104817 104 144 104

901 200 160

920 196 240 200

1010 256 208

1020 - 11203 100 160 104

RDTEX | Latch internals | RUOUG Seminar

x$ksmfsv ndash list of all fixed SGA variables983123E983116EC983124 DI983123983124I983118C983124 983147983155983149983142983155983155983145983162

F983122983119983117 983160$983147983155983149983142983155983158

983127HE983122E 983147983155983149983142983155983156983161983152 = 983147983155983148983148983156

Latch structure was bigger in 101 due to additional latch statistics

26

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2782

ORACLE LATCH IS NOT JUST A SINGLE MEMORY LOCATION

bull Before 11g Value of first latch byte (word for shared latches) wasused to determine latch state

0x00 ndash latch is free

0xFF ndash exclusive latch is busy Was 0x01 in Oracle 7

0x010x02hellip - shared latch holding by 12 hellip processes simultaneously

0x20000000 | pid - shared latch holding exclusively

bull In 11g first latch word shows the pid of the latch holder

0x00 ndash latch is free0x12 ndash Oracle process with pid 18 holds the exclusive latch

RDTEX | Latch internals | RUOUG Seminar

27

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2882

BUSY ORACLE 11G LATCH IN MEMORY

bull 112 Exclusive latch (32-bit platform)

bull 112 Shared latch EXCLUSIVE mode

RDTEX | Latch internals | RUOUG Seminar

SQLgt oradebug peek 200222A0 24

[200222A0 200222B8) = 00000016 00000001 000001D0 00000007

pid^ gets latch level

SQLgt oradebug peek 0xampladdr 24

[2000C814 2000C82C) = 2000001F 0000007C 0000007E 00000006 pid^ gets latch level

983139983151983150983156 983155983153983148983152983148983157983155 983150983151983148983151983143 9831409831419831499831519831350128

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2982

LATCH FIXED TABLES

bull Before Oracle 11g all the latches were externalized to SQL inv$latch (x$ksllt)

bull In 11g parent (fixed SGA) latches are externalized inv$latch_parent (x$kslltr_parent)

bull In 11g child (allocated on startup) latches are externalized in

v$latch_children (x$kslltr_children) bull Oracle 11g v$latch (x$kslltr) merges these data Beware select

from x$kslltr acquires all parent latches and their children

bull Only part of latch structure is visible in x$ksllthellip fixed tables ndash kslltwhr kslltwhy ndash ldquowhererdquo and ldquowhyrdquo latch was acquired last time

ndash kslltwgt hellip - statistics

ndash kslltlvl hellip - attributes

ndash kslltwtt ndash latch wait time

RDTEX | Latch internals | RUOUG Seminar

29

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3082

CUMULATIVE V$LATCH STATISTIC COUNTERS

RDTEX | Latch internals | RUOUG Seminar

Statistics x$ksllt When and how it increments

GETS kslltwgt ldquo++rdquo after wait mode latch get

MISSES kslltwff ldquo++rdquo after wait get if it was missed

SLEEPS kslltwsl ldquo+number_of_sleepsrdquo during get

SPIN_GETS ksllthst0 ldquo++rdquo if get was missed but not slept

WAIT_TIME kslltwtt ldquo+wait_timerdquo after latch get

IMMEDIATE_GETS kslltngt ldquo++rdquo after nowait mode latch get Is notincremented on miss

IMMEDIATE_MISSES kslltnfa ldquo++rdquo if nowait mode get was missed

30

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3182

MANY STATISTICS WERE LOST IN 102

In 102 Oracle made obsolete many useful latch statistics

Latch size was decreased and the following fields always contain 0now in v$latch

ndash v$latchwaiters_woken

ndash v$latchwaits_holding_latch

ndash v$latchsleep1hellipsleep10

ndash and many undocumented statistics

This was mentioned in 102 Reference

RDTEX | Latch internals | RUOUG Seminar

31

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3282

X$KSLLD - LATCH DESCRIPTORS

bull Flags and attributes describing each latch are stored in kslldt arrayof kslld structures

bull [K]ernel [S]ervice [L]atch [L]ock [D]escriptor

bull Some but not all of kslld fields are externalized via x$kslld andv$latchname

bull Located in the PGA

RDTEX | Latch internals | RUOUG Seminar

32

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382

LATCH ATTRIBUTES

bull Each latch have at least the following attributes in kslldt

ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared

ndash PAR Is the latch Solitary or Parent for the family of child latches

ndash G2C Can two child latches be simultaneously requested in wait mode

using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92

ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above

ndash Level 0-14 Used to prevent latch deadlocks

ndash Class 0-7 Spin and wait class assigned to the latch 92 and above

RDTEX | Latch internals | RUOUG Seminar

33

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482

LATCH ATTRIBUTES BY ORACLE VERSION

RDTEX | Latch internals | RUOUG Seminar

11203 553 154 72 - 6 93

Oracle Number of latches PAR G2C LNG UFS SHARED

7340 53 14 2 3 - -

8063 80 21 7 3 - 38174 152 48 19 4 - 9

9208 242 79 37 - - 19

10202 385 114 55 - 4 47

10203 388 117 58 - 4 48

10204 394 117 59 - 4 50

11106 496 145 67 - 6 81

11107 502 145 67 - 6 83

11201 535 149 70 - 6 86

11202 551 153 72 - 6 91

34

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582

LATCH LEVELS

bull Every latch has a level in the range 0-14 to avoid deadlocks

bull Process can request a latch X in wait mode after obtaining latch Y if and

only if level X gt level Y

bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y

bull If these rules are broken the Oracle process raises

ORA-600[504] - trying to obtain another latch at incompatible level Note 281041

ORA-600[525] - trying to obtain another latch at the same level Note 1388881

ORA-600[526] - latch child order violation Note 1388891

RDTEX | Latch internals | RUOUG Seminar

35

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682

PARENT AND CHILD LATCHES

bull Latch can be a parent child or solitary latch

bull Both parent and child latches share the same latch name

bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss

bull Child latches are allocated during instance startup

bull The parent latch can be gotten independently

bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s

RDTEX | Latch internals | RUOUG Seminar

(latch info) wait_event=0 bits=10

holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl

Context saved from call 0

state=busy

36

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782

LATCH TREES

ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches

Direct SGA access program output for 9206 instance with too smallshared pool

RDTEX | Latch internals | RUOUG Seminar

ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1

waiter ospid 129 sid 72 pid 45

holding a154b7120 library cache (157) level=5 child=17 kglupc child

waiter ospid 18255 sid 65 pid 930

waiter ospid 6690 sid 554 pid 1654

waiter ospid 4685 sid 879 pid 1034

hellip

waiter ospid 29749 sid 180 pid 155

holding a154b7db8 library cache (157) level=5 child=4 kglupc child

waiter ospid 13104 sid 281 pid 220

waiter ospid 24089 sid 565 pid 636

waiter ospid 25002 sid 621 pid 1481

waiter ospid 16930 sid 1046 pid 783hellip

37

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882

LATCH CLASSES

bull Classes with different spin and wait policies

bull Up to 8 classes By default all the latches belong to class 0

bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2

bull Latch can be assigned to class by its number For example

ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000

ndash _LATCH_CLASSES=46

bull Will be discussed later

RDTEX | Latch internals | RUOUG Seminar

SQLgt select from x$ksllclass

INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4

---- ------ ----- -------- ------ ------ ------ ------ ------

0 20000 0 1 8000 8000 8000 8000 8000

hellip

6 100 12 1000 2000 3000 4000 4000 4000

38

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982

LATCH ACQUISITION IN NO-WAIT MODE

bull kslgetl(laddr0whywhere) ndash Exclusive latch get

bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep

ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches

ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches

bull IMMEDIATE_ statistics are updated

bull Used when many candidate latches are available to bypass the

latch level rule or as part of CBC chain examination

RDTEX | Latch internals | RUOUG Seminar

sqlplus nolog demo_03sql39

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082

LATCH WAITS

RDTEX | Latch internals | RUOUG Seminar 40

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182

WAITING FOR THE LATCH

RDTEX | Latch internals | RUOUG Seminar

983123 G A

C983120983125 1 C983120983125 2

983120983154983151983139983141983155983155 A

983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)

983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144

983116983137983156983139983144

983120983154983151983139983141983155983155 B

41

RDTEX | L h i l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282

LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION

bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries

again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of

times

bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep

bull A sleep corresponds to a wait for the latch free wait event

bull Initially a process sleeps for 001 s This time is doubled in every

subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip

RDTEX | Latch internals | RUOUG Seminar

Still did not found any versionwhich used this

May be it was lt= Oracle 6

42

RDTEX | L t h i t l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382

LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81

Latch gets with wait (kslgetl(laddress1hellip))

ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times

ndash Sleep on latch free event with exponential backoff

ndash Repeat

RDTEX | Latch internals | RUOUG Seminar

43

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482

8I LATCH SPIN CODE FLOW USING DTRACE

RDTEX | Latch internals | RUOUG Seminar

bull hellip Event 10046 trace

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2

983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29

983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270

sqlplus nolog demo_04_8sql 29

44

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582

POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF

bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec

RDTEX | Latch internals | RUOUG Seminar

12 ]2 )1[(

minus= +wait N

timeout

45

bull Most waits were for nothing ndash latch already was free

bull Latch utilization can not be high

bull Unnecessary spins provoke CPU thrashing

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682

92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE

sqlplus nolog demo_05sql 45

RDTEX | Latch internals | RUOUG Seminar

kslgetl(0x50006318 1)

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt kslges(0x50006318 ) -wait latch get

-gt skgslsgts(0x50006318 ) -spin latch get

-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT

-gt kskthbwt(0x0)

-gt kslwlmod() - set up Wait List

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt skgpwwait -sleep latch getsemop(11 17-10 1)

983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140

46

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782

RELIABLE LATCH WAITS

bull Hidden latch wait revolution that we missed Latch relies on waitposting

bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout

bull Exclusive latch spins 20000 cycles by default

bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock

bull Latches assigned to non-default class wait until timeout

bull Version 90 was transient and used mix of backoff and wait posting

This is why _LATCH_WAIT_POSTING parameter disappeared in9i

bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin

RDTEX | Latch internals | RUOUG Seminar

47

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882

IMAGINE THE TIME MICROSCOPE

| a a s | a

Reality One million zoomed time

1 us 1 sec

1 ms 17 min1 sec 115 days

Light speed (300000 kmsec) Sonic speed (300 msec)

CPU tick (2 GHz) 00005 sec

Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec

Max spin time for shared latch (~220005 ticks) ~10 sec

Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec

Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec

OS context switch time (10us-1ms) 10sec-17min

48

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982

LATCH WAITS UNDER TIME MICROSCOPE

Old 8i latch algorithmSpin for latch during 5 seconds

Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min

hellipSpin again for 5 seconds

Sleep again for 23 days (2 sec) hellip

New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)

If spin not succeeded go to sleep until post

According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)

| |

49

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082

_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES

bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and

_LATCH_CLASS_0 parameters

bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000

bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter

bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch

50

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182

TIMED LATCH WAIT POSTING

bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel

bull This can lead to instance hang ndash process will never be woken up

bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout

bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem

bull Generic AIX ports always use timed latch wait posting

bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux

51

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282

FINE GRAIN LATCH TUNING

bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting

bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7

bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo

microseconds using pollsys() (not semtimedop()) system call

ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times

Loop up to ldquoSpinsrdquo cycles

Yield CPU using yield() (or sched_yield())

ndash Sleep for ldquoSleepXrdquo usecs

ndash Then spin again hellip

52

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382

TRACES OF NONDEFAULT CLASS LATCH

983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152

991270

9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000

991270 E983158983141983150983156 10046 983156983154983137983139983141

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482

PERFORMANCE OF LATCH CLASSES

Row cache objects latch contention testcase on ODA (24 SMT)54

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582

IS THE _SPIN_COUNT REALLY STATIC

bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch

waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used

ldquocache buffers chainrdquo latch contention

bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass

(_LATCH_CLASS_X) By default ndash 20000

bull Shared latch spin can be tuned by _SPIN_COUNT value By

default ndash 2000 However this is not trivial

55

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682

SHARED LATCH BEHAVE LIKE ENQUEUE

bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80

bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests

bull This blocking achieved by special 0x40000000 bit in the latch

value This bit is a flag that some incompatible latch acquisitionis in progress

bull Queued processes will be woken up one by one on latch release

bull X mode latch gets effectively serialize the shared latch

bull See my blog for description of experiments

56

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782

TRACE OF SHARED LATCH WAIT

Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the

latch wait list and then spins again

983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141

983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156

983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983147983155983148983159983148983149983151983140()

983147983155983148983155983147983143983155(098316050009BAC )

983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140

983155983141983149983151983152(27)

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882

SHARED LATCH ACQUISITION

bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned

bull X mode shared latch get spins by default up to 4000 cycles

bull S mode does not spin at all (or spins in unknown way)

S mode get X mode get

Held in S mode Compatible 2_SPIN_COUNT

Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT

58

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982

LATCH RELEASE

bull Free the latch ndash kslfre(laddr)

bull Oracle process releases the latch nonatomically

bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process

bull This requires less bus invalidation and ensures propagation of latch

release to other local cachesbull Not fair policy Spinners on the local CPU board have the

preference

bull Then process posts the first process in the list of waiters

59

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082

LATCH CONTENTION

RDTEX | Latch internals | RUOUG Seminar60

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182

LATCH CONTENTION

bull Occurs when a latch is required by several processes at thesame time

bull Mostly due to bugs or abnormal application behavior

bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp

bull To diagnose the latch contention we need to investigate thelatch statistics [8]

61

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282

DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS

Latch requests arrival rate

Immediate gets efficiency

Latch sleeps ratio

Latch wait time per second

Latch spin efficiencymisses

getsspin∆

∆= _σ

misses

sleepsK

∆=

time

gets

∆=λ

gets

misses

∆= ρ

time

timewait W

∆=

_

Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics

62

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382

SAMPLED AVERAGES

By sampling of x$ksupr we can estimate

ndash Length of latch wait queue

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS

ndash Number of spinning processes in 10g

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS

Sampling of v$lathholder estimates latch utilization

Using DTrace we can measure

ndash Latch acquisition and holding times

ndash Actual spin count

w L

s N

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482

DERIVED LATCH STATISTICS

Latch utilization (PASTA)

Average holding time

Length of latch wait list

Recurrent sleeps ratio

Latch acquisition time

time

timeholdinglatchU

∆=asymp

__ ρ

Re_100

___

questsGet

TimeSnap MissGet Pct S

lowast==

λ

W L =

κ

κ σ 1minus+

)(1

W N T saq += minusλ

64

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 20: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2082

LIST OF LATCH ldquoWHERErdquo LOCATIONS

X$KSLLW ndash List of latch [W]here Part of v$latch_misses

ndash ksllwnam ndash Kernel function name(ldquokghupr1rdquo ldquoksqdelrdquo etc) ndash ksllwlbl ndash Description of ldquowhyrdquo context

Join this code on indx to

ndash x$ksuprlatksulawhr (v$latchholder)ndash ldquowhererdquo value that wasused by latch holder

ndash x$kslltkslltwhr (v$latch) ndash ldquowhererdquo that was used by lastsuccessful latch acquisition

ndash x$ksuprksllawer (v$process) ndash ldquowhererdquo that was used byprocess is being waiting for latch

ndash x$kslwscindx (v$latch_misses) ndash latch sleep statistics by

ldquowhererdquo

RDTEX | Latch internals | RUOUG Seminar

20

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2182

V$LATCH_MISSES

bull Based on X$KSLWSC - No[W]ait and [S]leep [C]ount stats byContext Documentation is rather obscure

bull The following counters are incremented on each latch sleep

sleep_count (ldquowhererdquo the latch was held)

wtr_slp_count (ldquowhererdquo the latch get was requested)

bull nwfail_count is incremented only for session which SID is equalto _LATCH_MISS_STAT_SID parameter

bull longhold_count is incremented for shared latches only when

someone held the latch at this ldquowhererdquo for the entire duration ofsleep

RDTEX | Latch internals | RUOUG Seminar

21

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2282

LATCH IS HOLDING BY PROCESS NOT SESSION

bull

RDTEX | Latch internals | RUOUG Seminar

983123983156983154983157983139983156 983147983155983157983152983154 983163

991270

983123983156983154983157983139983156 983147983155983148983148983137983163983147983155983148983148983156 98314798315598314898314898313798314898313798315698313114983133

983165

991270983165

983120983154983151983139983141983155983155 983142983145983160983141983140 983137983154983154983137983161

983158$983152983154983151983139983141983155983155 983085gt 983160$983147983155983157983152983154

983123983156983154983157983139983156 983147983155983148983148983156983163

991270

983165

L983145983155983156 983151983142 983137983148983148 983148983137983156983139983144983141983155

983158$983148983137983156983139983144 983085gt983160$983147983155983148983148983156

Each process has an array of references to the latches it is holding

22

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2382

KSLLA IS HIDDEN IN V$PROCESS

bull The process latching info is the kslla structure embedded in theprocess state object

bull Some kslla fields were externalized to SQL

bull kslla contains array of latches holding by process at each level andinformation about the state of any current attempt to acquire a

latch Not externalized directly

bull When you query v$latchholder x$ksuprlat Oracle scansv$processx$ksupr in order to find all the processes that are

holding the latchesbull Tanel Poder uses high frequency sampling of v$latchholder to

systematically troubleshoot latch contention

RDTEX | Latch internals | RUOUG Seminar

23

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2482

X$KSUPRKSLLA FIELDS INSTRUMENT THE LATCH GET

bull ksllalaq ndash address of latch acquiring Populated during immediateget (and spin before 11g)

bull ksllawat - latch being waited for This is v$processlatchwait

bull ksllawhy ndash ldquowhyrdquo for the latch being waited for

bull ksllawere ndash ldquowhererdquo for the latch being waited for

bull ksllalow ndash bit array of levels of currently holding latches

bull ksllaspn - latch this process is spinning on v$processlatchspinNot populated since 81

bull ksllaps - inter-process post statistics

RDTEX | Latch internals | RUOUG Seminar

24

Generic ldquolatch freerdquo and 27 latch specific ldquolatch free rdquo wait events areassociated with sleep only Spin is invisible to Oracle Wait Interface

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2582

THE LATCH STRUCTURE ndash KSLLT

RDTEX | Latch internals | RUOUG Seminar

struct ksllt

ltLatchgt

ldquowhererdquo and ldquowhyrdquo

Level latch class other attributes

Statistics

Latch wait list header

hellip

25

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2682

LATCH SIZE BY VERSION

nix 32bit nix 64bit Windows 32bit

734 92 - 120

806 104 - 104817 104 144 104

901 200 160

920 196 240 200

1010 256 208

1020 - 11203 100 160 104

RDTEX | Latch internals | RUOUG Seminar

x$ksmfsv ndash list of all fixed SGA variables983123E983116EC983124 DI983123983124I983118C983124 983147983155983149983142983155983155983145983162

F983122983119983117 983160$983147983155983149983142983155983158

983127HE983122E 983147983155983149983142983155983156983161983152 = 983147983155983148983148983156

Latch structure was bigger in 101 due to additional latch statistics

26

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2782

ORACLE LATCH IS NOT JUST A SINGLE MEMORY LOCATION

bull Before 11g Value of first latch byte (word for shared latches) wasused to determine latch state

0x00 ndash latch is free

0xFF ndash exclusive latch is busy Was 0x01 in Oracle 7

0x010x02hellip - shared latch holding by 12 hellip processes simultaneously

0x20000000 | pid - shared latch holding exclusively

bull In 11g first latch word shows the pid of the latch holder

0x00 ndash latch is free0x12 ndash Oracle process with pid 18 holds the exclusive latch

RDTEX | Latch internals | RUOUG Seminar

27

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2882

BUSY ORACLE 11G LATCH IN MEMORY

bull 112 Exclusive latch (32-bit platform)

bull 112 Shared latch EXCLUSIVE mode

RDTEX | Latch internals | RUOUG Seminar

SQLgt oradebug peek 200222A0 24

[200222A0 200222B8) = 00000016 00000001 000001D0 00000007

pid^ gets latch level

SQLgt oradebug peek 0xampladdr 24

[2000C814 2000C82C) = 2000001F 0000007C 0000007E 00000006 pid^ gets latch level

983139983151983150983156 983155983153983148983152983148983157983155 983150983151983148983151983143 9831409831419831499831519831350128

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2982

LATCH FIXED TABLES

bull Before Oracle 11g all the latches were externalized to SQL inv$latch (x$ksllt)

bull In 11g parent (fixed SGA) latches are externalized inv$latch_parent (x$kslltr_parent)

bull In 11g child (allocated on startup) latches are externalized in

v$latch_children (x$kslltr_children) bull Oracle 11g v$latch (x$kslltr) merges these data Beware select

from x$kslltr acquires all parent latches and their children

bull Only part of latch structure is visible in x$ksllthellip fixed tables ndash kslltwhr kslltwhy ndash ldquowhererdquo and ldquowhyrdquo latch was acquired last time

ndash kslltwgt hellip - statistics

ndash kslltlvl hellip - attributes

ndash kslltwtt ndash latch wait time

RDTEX | Latch internals | RUOUG Seminar

29

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3082

CUMULATIVE V$LATCH STATISTIC COUNTERS

RDTEX | Latch internals | RUOUG Seminar

Statistics x$ksllt When and how it increments

GETS kslltwgt ldquo++rdquo after wait mode latch get

MISSES kslltwff ldquo++rdquo after wait get if it was missed

SLEEPS kslltwsl ldquo+number_of_sleepsrdquo during get

SPIN_GETS ksllthst0 ldquo++rdquo if get was missed but not slept

WAIT_TIME kslltwtt ldquo+wait_timerdquo after latch get

IMMEDIATE_GETS kslltngt ldquo++rdquo after nowait mode latch get Is notincremented on miss

IMMEDIATE_MISSES kslltnfa ldquo++rdquo if nowait mode get was missed

30

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3182

MANY STATISTICS WERE LOST IN 102

In 102 Oracle made obsolete many useful latch statistics

Latch size was decreased and the following fields always contain 0now in v$latch

ndash v$latchwaiters_woken

ndash v$latchwaits_holding_latch

ndash v$latchsleep1hellipsleep10

ndash and many undocumented statistics

This was mentioned in 102 Reference

RDTEX | Latch internals | RUOUG Seminar

31

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3282

X$KSLLD - LATCH DESCRIPTORS

bull Flags and attributes describing each latch are stored in kslldt arrayof kslld structures

bull [K]ernel [S]ervice [L]atch [L]ock [D]escriptor

bull Some but not all of kslld fields are externalized via x$kslld andv$latchname

bull Located in the PGA

RDTEX | Latch internals | RUOUG Seminar

32

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382

LATCH ATTRIBUTES

bull Each latch have at least the following attributes in kslldt

ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared

ndash PAR Is the latch Solitary or Parent for the family of child latches

ndash G2C Can two child latches be simultaneously requested in wait mode

using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92

ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above

ndash Level 0-14 Used to prevent latch deadlocks

ndash Class 0-7 Spin and wait class assigned to the latch 92 and above

RDTEX | Latch internals | RUOUG Seminar

33

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482

LATCH ATTRIBUTES BY ORACLE VERSION

RDTEX | Latch internals | RUOUG Seminar

11203 553 154 72 - 6 93

Oracle Number of latches PAR G2C LNG UFS SHARED

7340 53 14 2 3 - -

8063 80 21 7 3 - 38174 152 48 19 4 - 9

9208 242 79 37 - - 19

10202 385 114 55 - 4 47

10203 388 117 58 - 4 48

10204 394 117 59 - 4 50

11106 496 145 67 - 6 81

11107 502 145 67 - 6 83

11201 535 149 70 - 6 86

11202 551 153 72 - 6 91

34

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582

LATCH LEVELS

bull Every latch has a level in the range 0-14 to avoid deadlocks

bull Process can request a latch X in wait mode after obtaining latch Y if and

only if level X gt level Y

bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y

bull If these rules are broken the Oracle process raises

ORA-600[504] - trying to obtain another latch at incompatible level Note 281041

ORA-600[525] - trying to obtain another latch at the same level Note 1388881

ORA-600[526] - latch child order violation Note 1388891

RDTEX | Latch internals | RUOUG Seminar

35

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682

PARENT AND CHILD LATCHES

bull Latch can be a parent child or solitary latch

bull Both parent and child latches share the same latch name

bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss

bull Child latches are allocated during instance startup

bull The parent latch can be gotten independently

bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s

RDTEX | Latch internals | RUOUG Seminar

(latch info) wait_event=0 bits=10

holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl

Context saved from call 0

state=busy

36

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782

LATCH TREES

ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches

Direct SGA access program output for 9206 instance with too smallshared pool

RDTEX | Latch internals | RUOUG Seminar

ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1

waiter ospid 129 sid 72 pid 45

holding a154b7120 library cache (157) level=5 child=17 kglupc child

waiter ospid 18255 sid 65 pid 930

waiter ospid 6690 sid 554 pid 1654

waiter ospid 4685 sid 879 pid 1034

hellip

waiter ospid 29749 sid 180 pid 155

holding a154b7db8 library cache (157) level=5 child=4 kglupc child

waiter ospid 13104 sid 281 pid 220

waiter ospid 24089 sid 565 pid 636

waiter ospid 25002 sid 621 pid 1481

waiter ospid 16930 sid 1046 pid 783hellip

37

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882

LATCH CLASSES

bull Classes with different spin and wait policies

bull Up to 8 classes By default all the latches belong to class 0

bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2

bull Latch can be assigned to class by its number For example

ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000

ndash _LATCH_CLASSES=46

bull Will be discussed later

RDTEX | Latch internals | RUOUG Seminar

SQLgt select from x$ksllclass

INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4

---- ------ ----- -------- ------ ------ ------ ------ ------

0 20000 0 1 8000 8000 8000 8000 8000

hellip

6 100 12 1000 2000 3000 4000 4000 4000

38

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982

LATCH ACQUISITION IN NO-WAIT MODE

bull kslgetl(laddr0whywhere) ndash Exclusive latch get

bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep

ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches

ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches

bull IMMEDIATE_ statistics are updated

bull Used when many candidate latches are available to bypass the

latch level rule or as part of CBC chain examination

RDTEX | Latch internals | RUOUG Seminar

sqlplus nolog demo_03sql39

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082

LATCH WAITS

RDTEX | Latch internals | RUOUG Seminar 40

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182

WAITING FOR THE LATCH

RDTEX | Latch internals | RUOUG Seminar

983123 G A

C983120983125 1 C983120983125 2

983120983154983151983139983141983155983155 A

983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)

983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144

983116983137983156983139983144

983120983154983151983139983141983155983155 B

41

RDTEX | L h i l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282

LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION

bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries

again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of

times

bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep

bull A sleep corresponds to a wait for the latch free wait event

bull Initially a process sleeps for 001 s This time is doubled in every

subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip

RDTEX | Latch internals | RUOUG Seminar

Still did not found any versionwhich used this

May be it was lt= Oracle 6

42

RDTEX | L t h i t l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382

LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81

Latch gets with wait (kslgetl(laddress1hellip))

ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times

ndash Sleep on latch free event with exponential backoff

ndash Repeat

RDTEX | Latch internals | RUOUG Seminar

43

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482

8I LATCH SPIN CODE FLOW USING DTRACE

RDTEX | Latch internals | RUOUG Seminar

bull hellip Event 10046 trace

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2

983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29

983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270

sqlplus nolog demo_04_8sql 29

44

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582

POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF

bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec

RDTEX | Latch internals | RUOUG Seminar

12 ]2 )1[(

minus= +wait N

timeout

45

bull Most waits were for nothing ndash latch already was free

bull Latch utilization can not be high

bull Unnecessary spins provoke CPU thrashing

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682

92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE

sqlplus nolog demo_05sql 45

RDTEX | Latch internals | RUOUG Seminar

kslgetl(0x50006318 1)

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt kslges(0x50006318 ) -wait latch get

-gt skgslsgts(0x50006318 ) -spin latch get

-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT

-gt kskthbwt(0x0)

-gt kslwlmod() - set up Wait List

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt skgpwwait -sleep latch getsemop(11 17-10 1)

983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140

46

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782

RELIABLE LATCH WAITS

bull Hidden latch wait revolution that we missed Latch relies on waitposting

bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout

bull Exclusive latch spins 20000 cycles by default

bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock

bull Latches assigned to non-default class wait until timeout

bull Version 90 was transient and used mix of backoff and wait posting

This is why _LATCH_WAIT_POSTING parameter disappeared in9i

bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin

RDTEX | Latch internals | RUOUG Seminar

47

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882

IMAGINE THE TIME MICROSCOPE

| a a s | a

Reality One million zoomed time

1 us 1 sec

1 ms 17 min1 sec 115 days

Light speed (300000 kmsec) Sonic speed (300 msec)

CPU tick (2 GHz) 00005 sec

Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec

Max spin time for shared latch (~220005 ticks) ~10 sec

Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec

Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec

OS context switch time (10us-1ms) 10sec-17min

48

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982

LATCH WAITS UNDER TIME MICROSCOPE

Old 8i latch algorithmSpin for latch during 5 seconds

Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min

hellipSpin again for 5 seconds

Sleep again for 23 days (2 sec) hellip

New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)

If spin not succeeded go to sleep until post

According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)

| |

49

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082

_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES

bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and

_LATCH_CLASS_0 parameters

bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000

bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter

bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch

50

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182

TIMED LATCH WAIT POSTING

bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel

bull This can lead to instance hang ndash process will never be woken up

bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout

bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem

bull Generic AIX ports always use timed latch wait posting

bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux

51

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282

FINE GRAIN LATCH TUNING

bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting

bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7

bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo

microseconds using pollsys() (not semtimedop()) system call

ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times

Loop up to ldquoSpinsrdquo cycles

Yield CPU using yield() (or sched_yield())

ndash Sleep for ldquoSleepXrdquo usecs

ndash Then spin again hellip

52

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382

TRACES OF NONDEFAULT CLASS LATCH

983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152

991270

9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000

991270 E983158983141983150983156 10046 983156983154983137983139983141

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482

PERFORMANCE OF LATCH CLASSES

Row cache objects latch contention testcase on ODA (24 SMT)54

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582

IS THE _SPIN_COUNT REALLY STATIC

bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch

waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used

ldquocache buffers chainrdquo latch contention

bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass

(_LATCH_CLASS_X) By default ndash 20000

bull Shared latch spin can be tuned by _SPIN_COUNT value By

default ndash 2000 However this is not trivial

55

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682

SHARED LATCH BEHAVE LIKE ENQUEUE

bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80

bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests

bull This blocking achieved by special 0x40000000 bit in the latch

value This bit is a flag that some incompatible latch acquisitionis in progress

bull Queued processes will be woken up one by one on latch release

bull X mode latch gets effectively serialize the shared latch

bull See my blog for description of experiments

56

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782

TRACE OF SHARED LATCH WAIT

Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the

latch wait list and then spins again

983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141

983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156

983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983147983155983148983159983148983149983151983140()

983147983155983148983155983147983143983155(098316050009BAC )

983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140

983155983141983149983151983152(27)

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882

SHARED LATCH ACQUISITION

bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned

bull X mode shared latch get spins by default up to 4000 cycles

bull S mode does not spin at all (or spins in unknown way)

S mode get X mode get

Held in S mode Compatible 2_SPIN_COUNT

Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT

58

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982

LATCH RELEASE

bull Free the latch ndash kslfre(laddr)

bull Oracle process releases the latch nonatomically

bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process

bull This requires less bus invalidation and ensures propagation of latch

release to other local cachesbull Not fair policy Spinners on the local CPU board have the

preference

bull Then process posts the first process in the list of waiters

59

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082

LATCH CONTENTION

RDTEX | Latch internals | RUOUG Seminar60

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182

LATCH CONTENTION

bull Occurs when a latch is required by several processes at thesame time

bull Mostly due to bugs or abnormal application behavior

bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp

bull To diagnose the latch contention we need to investigate thelatch statistics [8]

61

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282

DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS

Latch requests arrival rate

Immediate gets efficiency

Latch sleeps ratio

Latch wait time per second

Latch spin efficiencymisses

getsspin∆

∆= _σ

misses

sleepsK

∆=

time

gets

∆=λ

gets

misses

∆= ρ

time

timewait W

∆=

_

Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics

62

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382

SAMPLED AVERAGES

By sampling of x$ksupr we can estimate

ndash Length of latch wait queue

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS

ndash Number of spinning processes in 10g

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS

Sampling of v$lathholder estimates latch utilization

Using DTrace we can measure

ndash Latch acquisition and holding times

ndash Actual spin count

w L

s N

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482

DERIVED LATCH STATISTICS

Latch utilization (PASTA)

Average holding time

Length of latch wait list

Recurrent sleeps ratio

Latch acquisition time

time

timeholdinglatchU

∆=asymp

__ ρ

Re_100

___

questsGet

TimeSnap MissGet Pct S

lowast==

λ

W L =

κ

κ σ 1minus+

)(1

W N T saq += minusλ

64

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 21: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2182

V$LATCH_MISSES

bull Based on X$KSLWSC - No[W]ait and [S]leep [C]ount stats byContext Documentation is rather obscure

bull The following counters are incremented on each latch sleep

sleep_count (ldquowhererdquo the latch was held)

wtr_slp_count (ldquowhererdquo the latch get was requested)

bull nwfail_count is incremented only for session which SID is equalto _LATCH_MISS_STAT_SID parameter

bull longhold_count is incremented for shared latches only when

someone held the latch at this ldquowhererdquo for the entire duration ofsleep

RDTEX | Latch internals | RUOUG Seminar

21

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2282

LATCH IS HOLDING BY PROCESS NOT SESSION

bull

RDTEX | Latch internals | RUOUG Seminar

983123983156983154983157983139983156 983147983155983157983152983154 983163

991270

983123983156983154983157983139983156 983147983155983148983148983137983163983147983155983148983148983156 98314798315598314898314898313798314898313798315698313114983133

983165

991270983165

983120983154983151983139983141983155983155 983142983145983160983141983140 983137983154983154983137983161

983158$983152983154983151983139983141983155983155 983085gt 983160$983147983155983157983152983154

983123983156983154983157983139983156 983147983155983148983148983156983163

991270

983165

L983145983155983156 983151983142 983137983148983148 983148983137983156983139983144983141983155

983158$983148983137983156983139983144 983085gt983160$983147983155983148983148983156

Each process has an array of references to the latches it is holding

22

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2382

KSLLA IS HIDDEN IN V$PROCESS

bull The process latching info is the kslla structure embedded in theprocess state object

bull Some kslla fields were externalized to SQL

bull kslla contains array of latches holding by process at each level andinformation about the state of any current attempt to acquire a

latch Not externalized directly

bull When you query v$latchholder x$ksuprlat Oracle scansv$processx$ksupr in order to find all the processes that are

holding the latchesbull Tanel Poder uses high frequency sampling of v$latchholder to

systematically troubleshoot latch contention

RDTEX | Latch internals | RUOUG Seminar

23

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2482

X$KSUPRKSLLA FIELDS INSTRUMENT THE LATCH GET

bull ksllalaq ndash address of latch acquiring Populated during immediateget (and spin before 11g)

bull ksllawat - latch being waited for This is v$processlatchwait

bull ksllawhy ndash ldquowhyrdquo for the latch being waited for

bull ksllawere ndash ldquowhererdquo for the latch being waited for

bull ksllalow ndash bit array of levels of currently holding latches

bull ksllaspn - latch this process is spinning on v$processlatchspinNot populated since 81

bull ksllaps - inter-process post statistics

RDTEX | Latch internals | RUOUG Seminar

24

Generic ldquolatch freerdquo and 27 latch specific ldquolatch free rdquo wait events areassociated with sleep only Spin is invisible to Oracle Wait Interface

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2582

THE LATCH STRUCTURE ndash KSLLT

RDTEX | Latch internals | RUOUG Seminar

struct ksllt

ltLatchgt

ldquowhererdquo and ldquowhyrdquo

Level latch class other attributes

Statistics

Latch wait list header

hellip

25

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2682

LATCH SIZE BY VERSION

nix 32bit nix 64bit Windows 32bit

734 92 - 120

806 104 - 104817 104 144 104

901 200 160

920 196 240 200

1010 256 208

1020 - 11203 100 160 104

RDTEX | Latch internals | RUOUG Seminar

x$ksmfsv ndash list of all fixed SGA variables983123E983116EC983124 DI983123983124I983118C983124 983147983155983149983142983155983155983145983162

F983122983119983117 983160$983147983155983149983142983155983158

983127HE983122E 983147983155983149983142983155983156983161983152 = 983147983155983148983148983156

Latch structure was bigger in 101 due to additional latch statistics

26

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2782

ORACLE LATCH IS NOT JUST A SINGLE MEMORY LOCATION

bull Before 11g Value of first latch byte (word for shared latches) wasused to determine latch state

0x00 ndash latch is free

0xFF ndash exclusive latch is busy Was 0x01 in Oracle 7

0x010x02hellip - shared latch holding by 12 hellip processes simultaneously

0x20000000 | pid - shared latch holding exclusively

bull In 11g first latch word shows the pid of the latch holder

0x00 ndash latch is free0x12 ndash Oracle process with pid 18 holds the exclusive latch

RDTEX | Latch internals | RUOUG Seminar

27

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2882

BUSY ORACLE 11G LATCH IN MEMORY

bull 112 Exclusive latch (32-bit platform)

bull 112 Shared latch EXCLUSIVE mode

RDTEX | Latch internals | RUOUG Seminar

SQLgt oradebug peek 200222A0 24

[200222A0 200222B8) = 00000016 00000001 000001D0 00000007

pid^ gets latch level

SQLgt oradebug peek 0xampladdr 24

[2000C814 2000C82C) = 2000001F 0000007C 0000007E 00000006 pid^ gets latch level

983139983151983150983156 983155983153983148983152983148983157983155 983150983151983148983151983143 9831409831419831499831519831350128

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2982

LATCH FIXED TABLES

bull Before Oracle 11g all the latches were externalized to SQL inv$latch (x$ksllt)

bull In 11g parent (fixed SGA) latches are externalized inv$latch_parent (x$kslltr_parent)

bull In 11g child (allocated on startup) latches are externalized in

v$latch_children (x$kslltr_children) bull Oracle 11g v$latch (x$kslltr) merges these data Beware select

from x$kslltr acquires all parent latches and their children

bull Only part of latch structure is visible in x$ksllthellip fixed tables ndash kslltwhr kslltwhy ndash ldquowhererdquo and ldquowhyrdquo latch was acquired last time

ndash kslltwgt hellip - statistics

ndash kslltlvl hellip - attributes

ndash kslltwtt ndash latch wait time

RDTEX | Latch internals | RUOUG Seminar

29

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3082

CUMULATIVE V$LATCH STATISTIC COUNTERS

RDTEX | Latch internals | RUOUG Seminar

Statistics x$ksllt When and how it increments

GETS kslltwgt ldquo++rdquo after wait mode latch get

MISSES kslltwff ldquo++rdquo after wait get if it was missed

SLEEPS kslltwsl ldquo+number_of_sleepsrdquo during get

SPIN_GETS ksllthst0 ldquo++rdquo if get was missed but not slept

WAIT_TIME kslltwtt ldquo+wait_timerdquo after latch get

IMMEDIATE_GETS kslltngt ldquo++rdquo after nowait mode latch get Is notincremented on miss

IMMEDIATE_MISSES kslltnfa ldquo++rdquo if nowait mode get was missed

30

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3182

MANY STATISTICS WERE LOST IN 102

In 102 Oracle made obsolete many useful latch statistics

Latch size was decreased and the following fields always contain 0now in v$latch

ndash v$latchwaiters_woken

ndash v$latchwaits_holding_latch

ndash v$latchsleep1hellipsleep10

ndash and many undocumented statistics

This was mentioned in 102 Reference

RDTEX | Latch internals | RUOUG Seminar

31

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3282

X$KSLLD - LATCH DESCRIPTORS

bull Flags and attributes describing each latch are stored in kslldt arrayof kslld structures

bull [K]ernel [S]ervice [L]atch [L]ock [D]escriptor

bull Some but not all of kslld fields are externalized via x$kslld andv$latchname

bull Located in the PGA

RDTEX | Latch internals | RUOUG Seminar

32

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382

LATCH ATTRIBUTES

bull Each latch have at least the following attributes in kslldt

ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared

ndash PAR Is the latch Solitary or Parent for the family of child latches

ndash G2C Can two child latches be simultaneously requested in wait mode

using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92

ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above

ndash Level 0-14 Used to prevent latch deadlocks

ndash Class 0-7 Spin and wait class assigned to the latch 92 and above

RDTEX | Latch internals | RUOUG Seminar

33

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482

LATCH ATTRIBUTES BY ORACLE VERSION

RDTEX | Latch internals | RUOUG Seminar

11203 553 154 72 - 6 93

Oracle Number of latches PAR G2C LNG UFS SHARED

7340 53 14 2 3 - -

8063 80 21 7 3 - 38174 152 48 19 4 - 9

9208 242 79 37 - - 19

10202 385 114 55 - 4 47

10203 388 117 58 - 4 48

10204 394 117 59 - 4 50

11106 496 145 67 - 6 81

11107 502 145 67 - 6 83

11201 535 149 70 - 6 86

11202 551 153 72 - 6 91

34

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582

LATCH LEVELS

bull Every latch has a level in the range 0-14 to avoid deadlocks

bull Process can request a latch X in wait mode after obtaining latch Y if and

only if level X gt level Y

bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y

bull If these rules are broken the Oracle process raises

ORA-600[504] - trying to obtain another latch at incompatible level Note 281041

ORA-600[525] - trying to obtain another latch at the same level Note 1388881

ORA-600[526] - latch child order violation Note 1388891

RDTEX | Latch internals | RUOUG Seminar

35

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682

PARENT AND CHILD LATCHES

bull Latch can be a parent child or solitary latch

bull Both parent and child latches share the same latch name

bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss

bull Child latches are allocated during instance startup

bull The parent latch can be gotten independently

bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s

RDTEX | Latch internals | RUOUG Seminar

(latch info) wait_event=0 bits=10

holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl

Context saved from call 0

state=busy

36

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782

LATCH TREES

ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches

Direct SGA access program output for 9206 instance with too smallshared pool

RDTEX | Latch internals | RUOUG Seminar

ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1

waiter ospid 129 sid 72 pid 45

holding a154b7120 library cache (157) level=5 child=17 kglupc child

waiter ospid 18255 sid 65 pid 930

waiter ospid 6690 sid 554 pid 1654

waiter ospid 4685 sid 879 pid 1034

hellip

waiter ospid 29749 sid 180 pid 155

holding a154b7db8 library cache (157) level=5 child=4 kglupc child

waiter ospid 13104 sid 281 pid 220

waiter ospid 24089 sid 565 pid 636

waiter ospid 25002 sid 621 pid 1481

waiter ospid 16930 sid 1046 pid 783hellip

37

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882

LATCH CLASSES

bull Classes with different spin and wait policies

bull Up to 8 classes By default all the latches belong to class 0

bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2

bull Latch can be assigned to class by its number For example

ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000

ndash _LATCH_CLASSES=46

bull Will be discussed later

RDTEX | Latch internals | RUOUG Seminar

SQLgt select from x$ksllclass

INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4

---- ------ ----- -------- ------ ------ ------ ------ ------

0 20000 0 1 8000 8000 8000 8000 8000

hellip

6 100 12 1000 2000 3000 4000 4000 4000

38

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982

LATCH ACQUISITION IN NO-WAIT MODE

bull kslgetl(laddr0whywhere) ndash Exclusive latch get

bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep

ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches

ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches

bull IMMEDIATE_ statistics are updated

bull Used when many candidate latches are available to bypass the

latch level rule or as part of CBC chain examination

RDTEX | Latch internals | RUOUG Seminar

sqlplus nolog demo_03sql39

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082

LATCH WAITS

RDTEX | Latch internals | RUOUG Seminar 40

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182

WAITING FOR THE LATCH

RDTEX | Latch internals | RUOUG Seminar

983123 G A

C983120983125 1 C983120983125 2

983120983154983151983139983141983155983155 A

983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)

983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144

983116983137983156983139983144

983120983154983151983139983141983155983155 B

41

RDTEX | L h i l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282

LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION

bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries

again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of

times

bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep

bull A sleep corresponds to a wait for the latch free wait event

bull Initially a process sleeps for 001 s This time is doubled in every

subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip

RDTEX | Latch internals | RUOUG Seminar

Still did not found any versionwhich used this

May be it was lt= Oracle 6

42

RDTEX | L t h i t l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382

LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81

Latch gets with wait (kslgetl(laddress1hellip))

ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times

ndash Sleep on latch free event with exponential backoff

ndash Repeat

RDTEX | Latch internals | RUOUG Seminar

43

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482

8I LATCH SPIN CODE FLOW USING DTRACE

RDTEX | Latch internals | RUOUG Seminar

bull hellip Event 10046 trace

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2

983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29

983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270

sqlplus nolog demo_04_8sql 29

44

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582

POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF

bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec

RDTEX | Latch internals | RUOUG Seminar

12 ]2 )1[(

minus= +wait N

timeout

45

bull Most waits were for nothing ndash latch already was free

bull Latch utilization can not be high

bull Unnecessary spins provoke CPU thrashing

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682

92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE

sqlplus nolog demo_05sql 45

RDTEX | Latch internals | RUOUG Seminar

kslgetl(0x50006318 1)

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt kslges(0x50006318 ) -wait latch get

-gt skgslsgts(0x50006318 ) -spin latch get

-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT

-gt kskthbwt(0x0)

-gt kslwlmod() - set up Wait List

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt skgpwwait -sleep latch getsemop(11 17-10 1)

983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140

46

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782

RELIABLE LATCH WAITS

bull Hidden latch wait revolution that we missed Latch relies on waitposting

bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout

bull Exclusive latch spins 20000 cycles by default

bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock

bull Latches assigned to non-default class wait until timeout

bull Version 90 was transient and used mix of backoff and wait posting

This is why _LATCH_WAIT_POSTING parameter disappeared in9i

bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin

RDTEX | Latch internals | RUOUG Seminar

47

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882

IMAGINE THE TIME MICROSCOPE

| a a s | a

Reality One million zoomed time

1 us 1 sec

1 ms 17 min1 sec 115 days

Light speed (300000 kmsec) Sonic speed (300 msec)

CPU tick (2 GHz) 00005 sec

Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec

Max spin time for shared latch (~220005 ticks) ~10 sec

Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec

Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec

OS context switch time (10us-1ms) 10sec-17min

48

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982

LATCH WAITS UNDER TIME MICROSCOPE

Old 8i latch algorithmSpin for latch during 5 seconds

Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min

hellipSpin again for 5 seconds

Sleep again for 23 days (2 sec) hellip

New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)

If spin not succeeded go to sleep until post

According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)

| |

49

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082

_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES

bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and

_LATCH_CLASS_0 parameters

bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000

bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter

bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch

50

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182

TIMED LATCH WAIT POSTING

bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel

bull This can lead to instance hang ndash process will never be woken up

bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout

bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem

bull Generic AIX ports always use timed latch wait posting

bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux

51

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282

FINE GRAIN LATCH TUNING

bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting

bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7

bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo

microseconds using pollsys() (not semtimedop()) system call

ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times

Loop up to ldquoSpinsrdquo cycles

Yield CPU using yield() (or sched_yield())

ndash Sleep for ldquoSleepXrdquo usecs

ndash Then spin again hellip

52

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382

TRACES OF NONDEFAULT CLASS LATCH

983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152

991270

9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000

991270 E983158983141983150983156 10046 983156983154983137983139983141

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482

PERFORMANCE OF LATCH CLASSES

Row cache objects latch contention testcase on ODA (24 SMT)54

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582

IS THE _SPIN_COUNT REALLY STATIC

bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch

waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used

ldquocache buffers chainrdquo latch contention

bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass

(_LATCH_CLASS_X) By default ndash 20000

bull Shared latch spin can be tuned by _SPIN_COUNT value By

default ndash 2000 However this is not trivial

55

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682

SHARED LATCH BEHAVE LIKE ENQUEUE

bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80

bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests

bull This blocking achieved by special 0x40000000 bit in the latch

value This bit is a flag that some incompatible latch acquisitionis in progress

bull Queued processes will be woken up one by one on latch release

bull X mode latch gets effectively serialize the shared latch

bull See my blog for description of experiments

56

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782

TRACE OF SHARED LATCH WAIT

Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the

latch wait list and then spins again

983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141

983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156

983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983147983155983148983159983148983149983151983140()

983147983155983148983155983147983143983155(098316050009BAC )

983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140

983155983141983149983151983152(27)

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882

SHARED LATCH ACQUISITION

bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned

bull X mode shared latch get spins by default up to 4000 cycles

bull S mode does not spin at all (or spins in unknown way)

S mode get X mode get

Held in S mode Compatible 2_SPIN_COUNT

Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT

58

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982

LATCH RELEASE

bull Free the latch ndash kslfre(laddr)

bull Oracle process releases the latch nonatomically

bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process

bull This requires less bus invalidation and ensures propagation of latch

release to other local cachesbull Not fair policy Spinners on the local CPU board have the

preference

bull Then process posts the first process in the list of waiters

59

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082

LATCH CONTENTION

RDTEX | Latch internals | RUOUG Seminar60

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182

LATCH CONTENTION

bull Occurs when a latch is required by several processes at thesame time

bull Mostly due to bugs or abnormal application behavior

bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp

bull To diagnose the latch contention we need to investigate thelatch statistics [8]

61

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282

DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS

Latch requests arrival rate

Immediate gets efficiency

Latch sleeps ratio

Latch wait time per second

Latch spin efficiencymisses

getsspin∆

∆= _σ

misses

sleepsK

∆=

time

gets

∆=λ

gets

misses

∆= ρ

time

timewait W

∆=

_

Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics

62

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382

SAMPLED AVERAGES

By sampling of x$ksupr we can estimate

ndash Length of latch wait queue

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS

ndash Number of spinning processes in 10g

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS

Sampling of v$lathholder estimates latch utilization

Using DTrace we can measure

ndash Latch acquisition and holding times

ndash Actual spin count

w L

s N

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482

DERIVED LATCH STATISTICS

Latch utilization (PASTA)

Average holding time

Length of latch wait list

Recurrent sleeps ratio

Latch acquisition time

time

timeholdinglatchU

∆=asymp

__ ρ

Re_100

___

questsGet

TimeSnap MissGet Pct S

lowast==

λ

W L =

κ

κ σ 1minus+

)(1

W N T saq += minusλ

64

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 22: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2282

LATCH IS HOLDING BY PROCESS NOT SESSION

bull

RDTEX | Latch internals | RUOUG Seminar

983123983156983154983157983139983156 983147983155983157983152983154 983163

991270

983123983156983154983157983139983156 983147983155983148983148983137983163983147983155983148983148983156 98314798315598314898314898313798314898313798315698313114983133

983165

991270983165

983120983154983151983139983141983155983155 983142983145983160983141983140 983137983154983154983137983161

983158$983152983154983151983139983141983155983155 983085gt 983160$983147983155983157983152983154

983123983156983154983157983139983156 983147983155983148983148983156983163

991270

983165

L983145983155983156 983151983142 983137983148983148 983148983137983156983139983144983141983155

983158$983148983137983156983139983144 983085gt983160$983147983155983148983148983156

Each process has an array of references to the latches it is holding

22

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2382

KSLLA IS HIDDEN IN V$PROCESS

bull The process latching info is the kslla structure embedded in theprocess state object

bull Some kslla fields were externalized to SQL

bull kslla contains array of latches holding by process at each level andinformation about the state of any current attempt to acquire a

latch Not externalized directly

bull When you query v$latchholder x$ksuprlat Oracle scansv$processx$ksupr in order to find all the processes that are

holding the latchesbull Tanel Poder uses high frequency sampling of v$latchholder to

systematically troubleshoot latch contention

RDTEX | Latch internals | RUOUG Seminar

23

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2482

X$KSUPRKSLLA FIELDS INSTRUMENT THE LATCH GET

bull ksllalaq ndash address of latch acquiring Populated during immediateget (and spin before 11g)

bull ksllawat - latch being waited for This is v$processlatchwait

bull ksllawhy ndash ldquowhyrdquo for the latch being waited for

bull ksllawere ndash ldquowhererdquo for the latch being waited for

bull ksllalow ndash bit array of levels of currently holding latches

bull ksllaspn - latch this process is spinning on v$processlatchspinNot populated since 81

bull ksllaps - inter-process post statistics

RDTEX | Latch internals | RUOUG Seminar

24

Generic ldquolatch freerdquo and 27 latch specific ldquolatch free rdquo wait events areassociated with sleep only Spin is invisible to Oracle Wait Interface

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2582

THE LATCH STRUCTURE ndash KSLLT

RDTEX | Latch internals | RUOUG Seminar

struct ksllt

ltLatchgt

ldquowhererdquo and ldquowhyrdquo

Level latch class other attributes

Statistics

Latch wait list header

hellip

25

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2682

LATCH SIZE BY VERSION

nix 32bit nix 64bit Windows 32bit

734 92 - 120

806 104 - 104817 104 144 104

901 200 160

920 196 240 200

1010 256 208

1020 - 11203 100 160 104

RDTEX | Latch internals | RUOUG Seminar

x$ksmfsv ndash list of all fixed SGA variables983123E983116EC983124 DI983123983124I983118C983124 983147983155983149983142983155983155983145983162

F983122983119983117 983160$983147983155983149983142983155983158

983127HE983122E 983147983155983149983142983155983156983161983152 = 983147983155983148983148983156

Latch structure was bigger in 101 due to additional latch statistics

26

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2782

ORACLE LATCH IS NOT JUST A SINGLE MEMORY LOCATION

bull Before 11g Value of first latch byte (word for shared latches) wasused to determine latch state

0x00 ndash latch is free

0xFF ndash exclusive latch is busy Was 0x01 in Oracle 7

0x010x02hellip - shared latch holding by 12 hellip processes simultaneously

0x20000000 | pid - shared latch holding exclusively

bull In 11g first latch word shows the pid of the latch holder

0x00 ndash latch is free0x12 ndash Oracle process with pid 18 holds the exclusive latch

RDTEX | Latch internals | RUOUG Seminar

27

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2882

BUSY ORACLE 11G LATCH IN MEMORY

bull 112 Exclusive latch (32-bit platform)

bull 112 Shared latch EXCLUSIVE mode

RDTEX | Latch internals | RUOUG Seminar

SQLgt oradebug peek 200222A0 24

[200222A0 200222B8) = 00000016 00000001 000001D0 00000007

pid^ gets latch level

SQLgt oradebug peek 0xampladdr 24

[2000C814 2000C82C) = 2000001F 0000007C 0000007E 00000006 pid^ gets latch level

983139983151983150983156 983155983153983148983152983148983157983155 983150983151983148983151983143 9831409831419831499831519831350128

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2982

LATCH FIXED TABLES

bull Before Oracle 11g all the latches were externalized to SQL inv$latch (x$ksllt)

bull In 11g parent (fixed SGA) latches are externalized inv$latch_parent (x$kslltr_parent)

bull In 11g child (allocated on startup) latches are externalized in

v$latch_children (x$kslltr_children) bull Oracle 11g v$latch (x$kslltr) merges these data Beware select

from x$kslltr acquires all parent latches and their children

bull Only part of latch structure is visible in x$ksllthellip fixed tables ndash kslltwhr kslltwhy ndash ldquowhererdquo and ldquowhyrdquo latch was acquired last time

ndash kslltwgt hellip - statistics

ndash kslltlvl hellip - attributes

ndash kslltwtt ndash latch wait time

RDTEX | Latch internals | RUOUG Seminar

29

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3082

CUMULATIVE V$LATCH STATISTIC COUNTERS

RDTEX | Latch internals | RUOUG Seminar

Statistics x$ksllt When and how it increments

GETS kslltwgt ldquo++rdquo after wait mode latch get

MISSES kslltwff ldquo++rdquo after wait get if it was missed

SLEEPS kslltwsl ldquo+number_of_sleepsrdquo during get

SPIN_GETS ksllthst0 ldquo++rdquo if get was missed but not slept

WAIT_TIME kslltwtt ldquo+wait_timerdquo after latch get

IMMEDIATE_GETS kslltngt ldquo++rdquo after nowait mode latch get Is notincremented on miss

IMMEDIATE_MISSES kslltnfa ldquo++rdquo if nowait mode get was missed

30

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3182

MANY STATISTICS WERE LOST IN 102

In 102 Oracle made obsolete many useful latch statistics

Latch size was decreased and the following fields always contain 0now in v$latch

ndash v$latchwaiters_woken

ndash v$latchwaits_holding_latch

ndash v$latchsleep1hellipsleep10

ndash and many undocumented statistics

This was mentioned in 102 Reference

RDTEX | Latch internals | RUOUG Seminar

31

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3282

X$KSLLD - LATCH DESCRIPTORS

bull Flags and attributes describing each latch are stored in kslldt arrayof kslld structures

bull [K]ernel [S]ervice [L]atch [L]ock [D]escriptor

bull Some but not all of kslld fields are externalized via x$kslld andv$latchname

bull Located in the PGA

RDTEX | Latch internals | RUOUG Seminar

32

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382

LATCH ATTRIBUTES

bull Each latch have at least the following attributes in kslldt

ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared

ndash PAR Is the latch Solitary or Parent for the family of child latches

ndash G2C Can two child latches be simultaneously requested in wait mode

using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92

ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above

ndash Level 0-14 Used to prevent latch deadlocks

ndash Class 0-7 Spin and wait class assigned to the latch 92 and above

RDTEX | Latch internals | RUOUG Seminar

33

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482

LATCH ATTRIBUTES BY ORACLE VERSION

RDTEX | Latch internals | RUOUG Seminar

11203 553 154 72 - 6 93

Oracle Number of latches PAR G2C LNG UFS SHARED

7340 53 14 2 3 - -

8063 80 21 7 3 - 38174 152 48 19 4 - 9

9208 242 79 37 - - 19

10202 385 114 55 - 4 47

10203 388 117 58 - 4 48

10204 394 117 59 - 4 50

11106 496 145 67 - 6 81

11107 502 145 67 - 6 83

11201 535 149 70 - 6 86

11202 551 153 72 - 6 91

34

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582

LATCH LEVELS

bull Every latch has a level in the range 0-14 to avoid deadlocks

bull Process can request a latch X in wait mode after obtaining latch Y if and

only if level X gt level Y

bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y

bull If these rules are broken the Oracle process raises

ORA-600[504] - trying to obtain another latch at incompatible level Note 281041

ORA-600[525] - trying to obtain another latch at the same level Note 1388881

ORA-600[526] - latch child order violation Note 1388891

RDTEX | Latch internals | RUOUG Seminar

35

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682

PARENT AND CHILD LATCHES

bull Latch can be a parent child or solitary latch

bull Both parent and child latches share the same latch name

bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss

bull Child latches are allocated during instance startup

bull The parent latch can be gotten independently

bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s

RDTEX | Latch internals | RUOUG Seminar

(latch info) wait_event=0 bits=10

holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl

Context saved from call 0

state=busy

36

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782

LATCH TREES

ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches

Direct SGA access program output for 9206 instance with too smallshared pool

RDTEX | Latch internals | RUOUG Seminar

ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1

waiter ospid 129 sid 72 pid 45

holding a154b7120 library cache (157) level=5 child=17 kglupc child

waiter ospid 18255 sid 65 pid 930

waiter ospid 6690 sid 554 pid 1654

waiter ospid 4685 sid 879 pid 1034

hellip

waiter ospid 29749 sid 180 pid 155

holding a154b7db8 library cache (157) level=5 child=4 kglupc child

waiter ospid 13104 sid 281 pid 220

waiter ospid 24089 sid 565 pid 636

waiter ospid 25002 sid 621 pid 1481

waiter ospid 16930 sid 1046 pid 783hellip

37

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882

LATCH CLASSES

bull Classes with different spin and wait policies

bull Up to 8 classes By default all the latches belong to class 0

bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2

bull Latch can be assigned to class by its number For example

ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000

ndash _LATCH_CLASSES=46

bull Will be discussed later

RDTEX | Latch internals | RUOUG Seminar

SQLgt select from x$ksllclass

INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4

---- ------ ----- -------- ------ ------ ------ ------ ------

0 20000 0 1 8000 8000 8000 8000 8000

hellip

6 100 12 1000 2000 3000 4000 4000 4000

38

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982

LATCH ACQUISITION IN NO-WAIT MODE

bull kslgetl(laddr0whywhere) ndash Exclusive latch get

bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep

ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches

ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches

bull IMMEDIATE_ statistics are updated

bull Used when many candidate latches are available to bypass the

latch level rule or as part of CBC chain examination

RDTEX | Latch internals | RUOUG Seminar

sqlplus nolog demo_03sql39

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082

LATCH WAITS

RDTEX | Latch internals | RUOUG Seminar 40

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182

WAITING FOR THE LATCH

RDTEX | Latch internals | RUOUG Seminar

983123 G A

C983120983125 1 C983120983125 2

983120983154983151983139983141983155983155 A

983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)

983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144

983116983137983156983139983144

983120983154983151983139983141983155983155 B

41

RDTEX | L h i l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282

LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION

bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries

again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of

times

bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep

bull A sleep corresponds to a wait for the latch free wait event

bull Initially a process sleeps for 001 s This time is doubled in every

subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip

RDTEX | Latch internals | RUOUG Seminar

Still did not found any versionwhich used this

May be it was lt= Oracle 6

42

RDTEX | L t h i t l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382

LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81

Latch gets with wait (kslgetl(laddress1hellip))

ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times

ndash Sleep on latch free event with exponential backoff

ndash Repeat

RDTEX | Latch internals | RUOUG Seminar

43

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482

8I LATCH SPIN CODE FLOW USING DTRACE

RDTEX | Latch internals | RUOUG Seminar

bull hellip Event 10046 trace

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2

983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29

983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270

sqlplus nolog demo_04_8sql 29

44

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582

POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF

bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec

RDTEX | Latch internals | RUOUG Seminar

12 ]2 )1[(

minus= +wait N

timeout

45

bull Most waits were for nothing ndash latch already was free

bull Latch utilization can not be high

bull Unnecessary spins provoke CPU thrashing

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682

92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE

sqlplus nolog demo_05sql 45

RDTEX | Latch internals | RUOUG Seminar

kslgetl(0x50006318 1)

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt kslges(0x50006318 ) -wait latch get

-gt skgslsgts(0x50006318 ) -spin latch get

-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT

-gt kskthbwt(0x0)

-gt kslwlmod() - set up Wait List

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt skgpwwait -sleep latch getsemop(11 17-10 1)

983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140

46

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782

RELIABLE LATCH WAITS

bull Hidden latch wait revolution that we missed Latch relies on waitposting

bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout

bull Exclusive latch spins 20000 cycles by default

bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock

bull Latches assigned to non-default class wait until timeout

bull Version 90 was transient and used mix of backoff and wait posting

This is why _LATCH_WAIT_POSTING parameter disappeared in9i

bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin

RDTEX | Latch internals | RUOUG Seminar

47

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882

IMAGINE THE TIME MICROSCOPE

| a a s | a

Reality One million zoomed time

1 us 1 sec

1 ms 17 min1 sec 115 days

Light speed (300000 kmsec) Sonic speed (300 msec)

CPU tick (2 GHz) 00005 sec

Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec

Max spin time for shared latch (~220005 ticks) ~10 sec

Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec

Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec

OS context switch time (10us-1ms) 10sec-17min

48

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982

LATCH WAITS UNDER TIME MICROSCOPE

Old 8i latch algorithmSpin for latch during 5 seconds

Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min

hellipSpin again for 5 seconds

Sleep again for 23 days (2 sec) hellip

New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)

If spin not succeeded go to sleep until post

According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)

| |

49

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082

_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES

bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and

_LATCH_CLASS_0 parameters

bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000

bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter

bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch

50

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182

TIMED LATCH WAIT POSTING

bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel

bull This can lead to instance hang ndash process will never be woken up

bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout

bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem

bull Generic AIX ports always use timed latch wait posting

bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux

51

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282

FINE GRAIN LATCH TUNING

bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting

bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7

bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo

microseconds using pollsys() (not semtimedop()) system call

ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times

Loop up to ldquoSpinsrdquo cycles

Yield CPU using yield() (or sched_yield())

ndash Sleep for ldquoSleepXrdquo usecs

ndash Then spin again hellip

52

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382

TRACES OF NONDEFAULT CLASS LATCH

983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152

991270

9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000

991270 E983158983141983150983156 10046 983156983154983137983139983141

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482

PERFORMANCE OF LATCH CLASSES

Row cache objects latch contention testcase on ODA (24 SMT)54

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582

IS THE _SPIN_COUNT REALLY STATIC

bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch

waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used

ldquocache buffers chainrdquo latch contention

bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass

(_LATCH_CLASS_X) By default ndash 20000

bull Shared latch spin can be tuned by _SPIN_COUNT value By

default ndash 2000 However this is not trivial

55

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682

SHARED LATCH BEHAVE LIKE ENQUEUE

bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80

bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests

bull This blocking achieved by special 0x40000000 bit in the latch

value This bit is a flag that some incompatible latch acquisitionis in progress

bull Queued processes will be woken up one by one on latch release

bull X mode latch gets effectively serialize the shared latch

bull See my blog for description of experiments

56

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782

TRACE OF SHARED LATCH WAIT

Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the

latch wait list and then spins again

983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141

983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156

983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983147983155983148983159983148983149983151983140()

983147983155983148983155983147983143983155(098316050009BAC )

983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140

983155983141983149983151983152(27)

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882

SHARED LATCH ACQUISITION

bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned

bull X mode shared latch get spins by default up to 4000 cycles

bull S mode does not spin at all (or spins in unknown way)

S mode get X mode get

Held in S mode Compatible 2_SPIN_COUNT

Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT

58

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982

LATCH RELEASE

bull Free the latch ndash kslfre(laddr)

bull Oracle process releases the latch nonatomically

bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process

bull This requires less bus invalidation and ensures propagation of latch

release to other local cachesbull Not fair policy Spinners on the local CPU board have the

preference

bull Then process posts the first process in the list of waiters

59

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082

LATCH CONTENTION

RDTEX | Latch internals | RUOUG Seminar60

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182

LATCH CONTENTION

bull Occurs when a latch is required by several processes at thesame time

bull Mostly due to bugs or abnormal application behavior

bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp

bull To diagnose the latch contention we need to investigate thelatch statistics [8]

61

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282

DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS

Latch requests arrival rate

Immediate gets efficiency

Latch sleeps ratio

Latch wait time per second

Latch spin efficiencymisses

getsspin∆

∆= _σ

misses

sleepsK

∆=

time

gets

∆=λ

gets

misses

∆= ρ

time

timewait W

∆=

_

Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics

62

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382

SAMPLED AVERAGES

By sampling of x$ksupr we can estimate

ndash Length of latch wait queue

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS

ndash Number of spinning processes in 10g

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS

Sampling of v$lathholder estimates latch utilization

Using DTrace we can measure

ndash Latch acquisition and holding times

ndash Actual spin count

w L

s N

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482

DERIVED LATCH STATISTICS

Latch utilization (PASTA)

Average holding time

Length of latch wait list

Recurrent sleeps ratio

Latch acquisition time

time

timeholdinglatchU

∆=asymp

__ ρ

Re_100

___

questsGet

TimeSnap MissGet Pct S

lowast==

λ

W L =

κ

κ σ 1minus+

)(1

W N T saq += minusλ

64

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 23: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2382

KSLLA IS HIDDEN IN V$PROCESS

bull The process latching info is the kslla structure embedded in theprocess state object

bull Some kslla fields were externalized to SQL

bull kslla contains array of latches holding by process at each level andinformation about the state of any current attempt to acquire a

latch Not externalized directly

bull When you query v$latchholder x$ksuprlat Oracle scansv$processx$ksupr in order to find all the processes that are

holding the latchesbull Tanel Poder uses high frequency sampling of v$latchholder to

systematically troubleshoot latch contention

RDTEX | Latch internals | RUOUG Seminar

23

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2482

X$KSUPRKSLLA FIELDS INSTRUMENT THE LATCH GET

bull ksllalaq ndash address of latch acquiring Populated during immediateget (and spin before 11g)

bull ksllawat - latch being waited for This is v$processlatchwait

bull ksllawhy ndash ldquowhyrdquo for the latch being waited for

bull ksllawere ndash ldquowhererdquo for the latch being waited for

bull ksllalow ndash bit array of levels of currently holding latches

bull ksllaspn - latch this process is spinning on v$processlatchspinNot populated since 81

bull ksllaps - inter-process post statistics

RDTEX | Latch internals | RUOUG Seminar

24

Generic ldquolatch freerdquo and 27 latch specific ldquolatch free rdquo wait events areassociated with sleep only Spin is invisible to Oracle Wait Interface

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2582

THE LATCH STRUCTURE ndash KSLLT

RDTEX | Latch internals | RUOUG Seminar

struct ksllt

ltLatchgt

ldquowhererdquo and ldquowhyrdquo

Level latch class other attributes

Statistics

Latch wait list header

hellip

25

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2682

LATCH SIZE BY VERSION

nix 32bit nix 64bit Windows 32bit

734 92 - 120

806 104 - 104817 104 144 104

901 200 160

920 196 240 200

1010 256 208

1020 - 11203 100 160 104

RDTEX | Latch internals | RUOUG Seminar

x$ksmfsv ndash list of all fixed SGA variables983123E983116EC983124 DI983123983124I983118C983124 983147983155983149983142983155983155983145983162

F983122983119983117 983160$983147983155983149983142983155983158

983127HE983122E 983147983155983149983142983155983156983161983152 = 983147983155983148983148983156

Latch structure was bigger in 101 due to additional latch statistics

26

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2782

ORACLE LATCH IS NOT JUST A SINGLE MEMORY LOCATION

bull Before 11g Value of first latch byte (word for shared latches) wasused to determine latch state

0x00 ndash latch is free

0xFF ndash exclusive latch is busy Was 0x01 in Oracle 7

0x010x02hellip - shared latch holding by 12 hellip processes simultaneously

0x20000000 | pid - shared latch holding exclusively

bull In 11g first latch word shows the pid of the latch holder

0x00 ndash latch is free0x12 ndash Oracle process with pid 18 holds the exclusive latch

RDTEX | Latch internals | RUOUG Seminar

27

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2882

BUSY ORACLE 11G LATCH IN MEMORY

bull 112 Exclusive latch (32-bit platform)

bull 112 Shared latch EXCLUSIVE mode

RDTEX | Latch internals | RUOUG Seminar

SQLgt oradebug peek 200222A0 24

[200222A0 200222B8) = 00000016 00000001 000001D0 00000007

pid^ gets latch level

SQLgt oradebug peek 0xampladdr 24

[2000C814 2000C82C) = 2000001F 0000007C 0000007E 00000006 pid^ gets latch level

983139983151983150983156 983155983153983148983152983148983157983155 983150983151983148983151983143 9831409831419831499831519831350128

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2982

LATCH FIXED TABLES

bull Before Oracle 11g all the latches were externalized to SQL inv$latch (x$ksllt)

bull In 11g parent (fixed SGA) latches are externalized inv$latch_parent (x$kslltr_parent)

bull In 11g child (allocated on startup) latches are externalized in

v$latch_children (x$kslltr_children) bull Oracle 11g v$latch (x$kslltr) merges these data Beware select

from x$kslltr acquires all parent latches and their children

bull Only part of latch structure is visible in x$ksllthellip fixed tables ndash kslltwhr kslltwhy ndash ldquowhererdquo and ldquowhyrdquo latch was acquired last time

ndash kslltwgt hellip - statistics

ndash kslltlvl hellip - attributes

ndash kslltwtt ndash latch wait time

RDTEX | Latch internals | RUOUG Seminar

29

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3082

CUMULATIVE V$LATCH STATISTIC COUNTERS

RDTEX | Latch internals | RUOUG Seminar

Statistics x$ksllt When and how it increments

GETS kslltwgt ldquo++rdquo after wait mode latch get

MISSES kslltwff ldquo++rdquo after wait get if it was missed

SLEEPS kslltwsl ldquo+number_of_sleepsrdquo during get

SPIN_GETS ksllthst0 ldquo++rdquo if get was missed but not slept

WAIT_TIME kslltwtt ldquo+wait_timerdquo after latch get

IMMEDIATE_GETS kslltngt ldquo++rdquo after nowait mode latch get Is notincremented on miss

IMMEDIATE_MISSES kslltnfa ldquo++rdquo if nowait mode get was missed

30

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3182

MANY STATISTICS WERE LOST IN 102

In 102 Oracle made obsolete many useful latch statistics

Latch size was decreased and the following fields always contain 0now in v$latch

ndash v$latchwaiters_woken

ndash v$latchwaits_holding_latch

ndash v$latchsleep1hellipsleep10

ndash and many undocumented statistics

This was mentioned in 102 Reference

RDTEX | Latch internals | RUOUG Seminar

31

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3282

X$KSLLD - LATCH DESCRIPTORS

bull Flags and attributes describing each latch are stored in kslldt arrayof kslld structures

bull [K]ernel [S]ervice [L]atch [L]ock [D]escriptor

bull Some but not all of kslld fields are externalized via x$kslld andv$latchname

bull Located in the PGA

RDTEX | Latch internals | RUOUG Seminar

32

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382

LATCH ATTRIBUTES

bull Each latch have at least the following attributes in kslldt

ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared

ndash PAR Is the latch Solitary or Parent for the family of child latches

ndash G2C Can two child latches be simultaneously requested in wait mode

using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92

ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above

ndash Level 0-14 Used to prevent latch deadlocks

ndash Class 0-7 Spin and wait class assigned to the latch 92 and above

RDTEX | Latch internals | RUOUG Seminar

33

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482

LATCH ATTRIBUTES BY ORACLE VERSION

RDTEX | Latch internals | RUOUG Seminar

11203 553 154 72 - 6 93

Oracle Number of latches PAR G2C LNG UFS SHARED

7340 53 14 2 3 - -

8063 80 21 7 3 - 38174 152 48 19 4 - 9

9208 242 79 37 - - 19

10202 385 114 55 - 4 47

10203 388 117 58 - 4 48

10204 394 117 59 - 4 50

11106 496 145 67 - 6 81

11107 502 145 67 - 6 83

11201 535 149 70 - 6 86

11202 551 153 72 - 6 91

34

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582

LATCH LEVELS

bull Every latch has a level in the range 0-14 to avoid deadlocks

bull Process can request a latch X in wait mode after obtaining latch Y if and

only if level X gt level Y

bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y

bull If these rules are broken the Oracle process raises

ORA-600[504] - trying to obtain another latch at incompatible level Note 281041

ORA-600[525] - trying to obtain another latch at the same level Note 1388881

ORA-600[526] - latch child order violation Note 1388891

RDTEX | Latch internals | RUOUG Seminar

35

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682

PARENT AND CHILD LATCHES

bull Latch can be a parent child or solitary latch

bull Both parent and child latches share the same latch name

bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss

bull Child latches are allocated during instance startup

bull The parent latch can be gotten independently

bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s

RDTEX | Latch internals | RUOUG Seminar

(latch info) wait_event=0 bits=10

holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl

Context saved from call 0

state=busy

36

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782

LATCH TREES

ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches

Direct SGA access program output for 9206 instance with too smallshared pool

RDTEX | Latch internals | RUOUG Seminar

ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1

waiter ospid 129 sid 72 pid 45

holding a154b7120 library cache (157) level=5 child=17 kglupc child

waiter ospid 18255 sid 65 pid 930

waiter ospid 6690 sid 554 pid 1654

waiter ospid 4685 sid 879 pid 1034

hellip

waiter ospid 29749 sid 180 pid 155

holding a154b7db8 library cache (157) level=5 child=4 kglupc child

waiter ospid 13104 sid 281 pid 220

waiter ospid 24089 sid 565 pid 636

waiter ospid 25002 sid 621 pid 1481

waiter ospid 16930 sid 1046 pid 783hellip

37

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882

LATCH CLASSES

bull Classes with different spin and wait policies

bull Up to 8 classes By default all the latches belong to class 0

bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2

bull Latch can be assigned to class by its number For example

ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000

ndash _LATCH_CLASSES=46

bull Will be discussed later

RDTEX | Latch internals | RUOUG Seminar

SQLgt select from x$ksllclass

INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4

---- ------ ----- -------- ------ ------ ------ ------ ------

0 20000 0 1 8000 8000 8000 8000 8000

hellip

6 100 12 1000 2000 3000 4000 4000 4000

38

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982

LATCH ACQUISITION IN NO-WAIT MODE

bull kslgetl(laddr0whywhere) ndash Exclusive latch get

bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep

ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches

ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches

bull IMMEDIATE_ statistics are updated

bull Used when many candidate latches are available to bypass the

latch level rule or as part of CBC chain examination

RDTEX | Latch internals | RUOUG Seminar

sqlplus nolog demo_03sql39

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082

LATCH WAITS

RDTEX | Latch internals | RUOUG Seminar 40

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182

WAITING FOR THE LATCH

RDTEX | Latch internals | RUOUG Seminar

983123 G A

C983120983125 1 C983120983125 2

983120983154983151983139983141983155983155 A

983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)

983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144

983116983137983156983139983144

983120983154983151983139983141983155983155 B

41

RDTEX | L h i l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282

LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION

bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries

again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of

times

bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep

bull A sleep corresponds to a wait for the latch free wait event

bull Initially a process sleeps for 001 s This time is doubled in every

subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip

RDTEX | Latch internals | RUOUG Seminar

Still did not found any versionwhich used this

May be it was lt= Oracle 6

42

RDTEX | L t h i t l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382

LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81

Latch gets with wait (kslgetl(laddress1hellip))

ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times

ndash Sleep on latch free event with exponential backoff

ndash Repeat

RDTEX | Latch internals | RUOUG Seminar

43

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482

8I LATCH SPIN CODE FLOW USING DTRACE

RDTEX | Latch internals | RUOUG Seminar

bull hellip Event 10046 trace

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2

983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29

983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270

sqlplus nolog demo_04_8sql 29

44

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582

POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF

bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec

RDTEX | Latch internals | RUOUG Seminar

12 ]2 )1[(

minus= +wait N

timeout

45

bull Most waits were for nothing ndash latch already was free

bull Latch utilization can not be high

bull Unnecessary spins provoke CPU thrashing

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682

92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE

sqlplus nolog demo_05sql 45

RDTEX | Latch internals | RUOUG Seminar

kslgetl(0x50006318 1)

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt kslges(0x50006318 ) -wait latch get

-gt skgslsgts(0x50006318 ) -spin latch get

-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT

-gt kskthbwt(0x0)

-gt kslwlmod() - set up Wait List

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt skgpwwait -sleep latch getsemop(11 17-10 1)

983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140

46

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782

RELIABLE LATCH WAITS

bull Hidden latch wait revolution that we missed Latch relies on waitposting

bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout

bull Exclusive latch spins 20000 cycles by default

bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock

bull Latches assigned to non-default class wait until timeout

bull Version 90 was transient and used mix of backoff and wait posting

This is why _LATCH_WAIT_POSTING parameter disappeared in9i

bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin

RDTEX | Latch internals | RUOUG Seminar

47

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882

IMAGINE THE TIME MICROSCOPE

| a a s | a

Reality One million zoomed time

1 us 1 sec

1 ms 17 min1 sec 115 days

Light speed (300000 kmsec) Sonic speed (300 msec)

CPU tick (2 GHz) 00005 sec

Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec

Max spin time for shared latch (~220005 ticks) ~10 sec

Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec

Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec

OS context switch time (10us-1ms) 10sec-17min

48

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982

LATCH WAITS UNDER TIME MICROSCOPE

Old 8i latch algorithmSpin for latch during 5 seconds

Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min

hellipSpin again for 5 seconds

Sleep again for 23 days (2 sec) hellip

New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)

If spin not succeeded go to sleep until post

According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)

| |

49

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082

_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES

bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and

_LATCH_CLASS_0 parameters

bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000

bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter

bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch

50

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182

TIMED LATCH WAIT POSTING

bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel

bull This can lead to instance hang ndash process will never be woken up

bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout

bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem

bull Generic AIX ports always use timed latch wait posting

bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux

51

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282

FINE GRAIN LATCH TUNING

bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting

bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7

bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo

microseconds using pollsys() (not semtimedop()) system call

ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times

Loop up to ldquoSpinsrdquo cycles

Yield CPU using yield() (or sched_yield())

ndash Sleep for ldquoSleepXrdquo usecs

ndash Then spin again hellip

52

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382

TRACES OF NONDEFAULT CLASS LATCH

983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152

991270

9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000

991270 E983158983141983150983156 10046 983156983154983137983139983141

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482

PERFORMANCE OF LATCH CLASSES

Row cache objects latch contention testcase on ODA (24 SMT)54

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582

IS THE _SPIN_COUNT REALLY STATIC

bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch

waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used

ldquocache buffers chainrdquo latch contention

bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass

(_LATCH_CLASS_X) By default ndash 20000

bull Shared latch spin can be tuned by _SPIN_COUNT value By

default ndash 2000 However this is not trivial

55

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682

SHARED LATCH BEHAVE LIKE ENQUEUE

bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80

bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests

bull This blocking achieved by special 0x40000000 bit in the latch

value This bit is a flag that some incompatible latch acquisitionis in progress

bull Queued processes will be woken up one by one on latch release

bull X mode latch gets effectively serialize the shared latch

bull See my blog for description of experiments

56

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782

TRACE OF SHARED LATCH WAIT

Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the

latch wait list and then spins again

983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141

983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156

983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983147983155983148983159983148983149983151983140()

983147983155983148983155983147983143983155(098316050009BAC )

983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140

983155983141983149983151983152(27)

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882

SHARED LATCH ACQUISITION

bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned

bull X mode shared latch get spins by default up to 4000 cycles

bull S mode does not spin at all (or spins in unknown way)

S mode get X mode get

Held in S mode Compatible 2_SPIN_COUNT

Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT

58

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982

LATCH RELEASE

bull Free the latch ndash kslfre(laddr)

bull Oracle process releases the latch nonatomically

bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process

bull This requires less bus invalidation and ensures propagation of latch

release to other local cachesbull Not fair policy Spinners on the local CPU board have the

preference

bull Then process posts the first process in the list of waiters

59

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082

LATCH CONTENTION

RDTEX | Latch internals | RUOUG Seminar60

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182

LATCH CONTENTION

bull Occurs when a latch is required by several processes at thesame time

bull Mostly due to bugs or abnormal application behavior

bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp

bull To diagnose the latch contention we need to investigate thelatch statistics [8]

61

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282

DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS

Latch requests arrival rate

Immediate gets efficiency

Latch sleeps ratio

Latch wait time per second

Latch spin efficiencymisses

getsspin∆

∆= _σ

misses

sleepsK

∆=

time

gets

∆=λ

gets

misses

∆= ρ

time

timewait W

∆=

_

Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics

62

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382

SAMPLED AVERAGES

By sampling of x$ksupr we can estimate

ndash Length of latch wait queue

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS

ndash Number of spinning processes in 10g

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS

Sampling of v$lathholder estimates latch utilization

Using DTrace we can measure

ndash Latch acquisition and holding times

ndash Actual spin count

w L

s N

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482

DERIVED LATCH STATISTICS

Latch utilization (PASTA)

Average holding time

Length of latch wait list

Recurrent sleeps ratio

Latch acquisition time

time

timeholdinglatchU

∆=asymp

__ ρ

Re_100

___

questsGet

TimeSnap MissGet Pct S

lowast==

λ

W L =

κ

κ σ 1minus+

)(1

W N T saq += minusλ

64

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 24: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2482

X$KSUPRKSLLA FIELDS INSTRUMENT THE LATCH GET

bull ksllalaq ndash address of latch acquiring Populated during immediateget (and spin before 11g)

bull ksllawat - latch being waited for This is v$processlatchwait

bull ksllawhy ndash ldquowhyrdquo for the latch being waited for

bull ksllawere ndash ldquowhererdquo for the latch being waited for

bull ksllalow ndash bit array of levels of currently holding latches

bull ksllaspn - latch this process is spinning on v$processlatchspinNot populated since 81

bull ksllaps - inter-process post statistics

RDTEX | Latch internals | RUOUG Seminar

24

Generic ldquolatch freerdquo and 27 latch specific ldquolatch free rdquo wait events areassociated with sleep only Spin is invisible to Oracle Wait Interface

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2582

THE LATCH STRUCTURE ndash KSLLT

RDTEX | Latch internals | RUOUG Seminar

struct ksllt

ltLatchgt

ldquowhererdquo and ldquowhyrdquo

Level latch class other attributes

Statistics

Latch wait list header

hellip

25

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2682

LATCH SIZE BY VERSION

nix 32bit nix 64bit Windows 32bit

734 92 - 120

806 104 - 104817 104 144 104

901 200 160

920 196 240 200

1010 256 208

1020 - 11203 100 160 104

RDTEX | Latch internals | RUOUG Seminar

x$ksmfsv ndash list of all fixed SGA variables983123E983116EC983124 DI983123983124I983118C983124 983147983155983149983142983155983155983145983162

F983122983119983117 983160$983147983155983149983142983155983158

983127HE983122E 983147983155983149983142983155983156983161983152 = 983147983155983148983148983156

Latch structure was bigger in 101 due to additional latch statistics

26

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2782

ORACLE LATCH IS NOT JUST A SINGLE MEMORY LOCATION

bull Before 11g Value of first latch byte (word for shared latches) wasused to determine latch state

0x00 ndash latch is free

0xFF ndash exclusive latch is busy Was 0x01 in Oracle 7

0x010x02hellip - shared latch holding by 12 hellip processes simultaneously

0x20000000 | pid - shared latch holding exclusively

bull In 11g first latch word shows the pid of the latch holder

0x00 ndash latch is free0x12 ndash Oracle process with pid 18 holds the exclusive latch

RDTEX | Latch internals | RUOUG Seminar

27

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2882

BUSY ORACLE 11G LATCH IN MEMORY

bull 112 Exclusive latch (32-bit platform)

bull 112 Shared latch EXCLUSIVE mode

RDTEX | Latch internals | RUOUG Seminar

SQLgt oradebug peek 200222A0 24

[200222A0 200222B8) = 00000016 00000001 000001D0 00000007

pid^ gets latch level

SQLgt oradebug peek 0xampladdr 24

[2000C814 2000C82C) = 2000001F 0000007C 0000007E 00000006 pid^ gets latch level

983139983151983150983156 983155983153983148983152983148983157983155 983150983151983148983151983143 9831409831419831499831519831350128

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2982

LATCH FIXED TABLES

bull Before Oracle 11g all the latches were externalized to SQL inv$latch (x$ksllt)

bull In 11g parent (fixed SGA) latches are externalized inv$latch_parent (x$kslltr_parent)

bull In 11g child (allocated on startup) latches are externalized in

v$latch_children (x$kslltr_children) bull Oracle 11g v$latch (x$kslltr) merges these data Beware select

from x$kslltr acquires all parent latches and their children

bull Only part of latch structure is visible in x$ksllthellip fixed tables ndash kslltwhr kslltwhy ndash ldquowhererdquo and ldquowhyrdquo latch was acquired last time

ndash kslltwgt hellip - statistics

ndash kslltlvl hellip - attributes

ndash kslltwtt ndash latch wait time

RDTEX | Latch internals | RUOUG Seminar

29

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3082

CUMULATIVE V$LATCH STATISTIC COUNTERS

RDTEX | Latch internals | RUOUG Seminar

Statistics x$ksllt When and how it increments

GETS kslltwgt ldquo++rdquo after wait mode latch get

MISSES kslltwff ldquo++rdquo after wait get if it was missed

SLEEPS kslltwsl ldquo+number_of_sleepsrdquo during get

SPIN_GETS ksllthst0 ldquo++rdquo if get was missed but not slept

WAIT_TIME kslltwtt ldquo+wait_timerdquo after latch get

IMMEDIATE_GETS kslltngt ldquo++rdquo after nowait mode latch get Is notincremented on miss

IMMEDIATE_MISSES kslltnfa ldquo++rdquo if nowait mode get was missed

30

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3182

MANY STATISTICS WERE LOST IN 102

In 102 Oracle made obsolete many useful latch statistics

Latch size was decreased and the following fields always contain 0now in v$latch

ndash v$latchwaiters_woken

ndash v$latchwaits_holding_latch

ndash v$latchsleep1hellipsleep10

ndash and many undocumented statistics

This was mentioned in 102 Reference

RDTEX | Latch internals | RUOUG Seminar

31

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3282

X$KSLLD - LATCH DESCRIPTORS

bull Flags and attributes describing each latch are stored in kslldt arrayof kslld structures

bull [K]ernel [S]ervice [L]atch [L]ock [D]escriptor

bull Some but not all of kslld fields are externalized via x$kslld andv$latchname

bull Located in the PGA

RDTEX | Latch internals | RUOUG Seminar

32

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382

LATCH ATTRIBUTES

bull Each latch have at least the following attributes in kslldt

ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared

ndash PAR Is the latch Solitary or Parent for the family of child latches

ndash G2C Can two child latches be simultaneously requested in wait mode

using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92

ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above

ndash Level 0-14 Used to prevent latch deadlocks

ndash Class 0-7 Spin and wait class assigned to the latch 92 and above

RDTEX | Latch internals | RUOUG Seminar

33

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482

LATCH ATTRIBUTES BY ORACLE VERSION

RDTEX | Latch internals | RUOUG Seminar

11203 553 154 72 - 6 93

Oracle Number of latches PAR G2C LNG UFS SHARED

7340 53 14 2 3 - -

8063 80 21 7 3 - 38174 152 48 19 4 - 9

9208 242 79 37 - - 19

10202 385 114 55 - 4 47

10203 388 117 58 - 4 48

10204 394 117 59 - 4 50

11106 496 145 67 - 6 81

11107 502 145 67 - 6 83

11201 535 149 70 - 6 86

11202 551 153 72 - 6 91

34

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582

LATCH LEVELS

bull Every latch has a level in the range 0-14 to avoid deadlocks

bull Process can request a latch X in wait mode after obtaining latch Y if and

only if level X gt level Y

bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y

bull If these rules are broken the Oracle process raises

ORA-600[504] - trying to obtain another latch at incompatible level Note 281041

ORA-600[525] - trying to obtain another latch at the same level Note 1388881

ORA-600[526] - latch child order violation Note 1388891

RDTEX | Latch internals | RUOUG Seminar

35

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682

PARENT AND CHILD LATCHES

bull Latch can be a parent child or solitary latch

bull Both parent and child latches share the same latch name

bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss

bull Child latches are allocated during instance startup

bull The parent latch can be gotten independently

bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s

RDTEX | Latch internals | RUOUG Seminar

(latch info) wait_event=0 bits=10

holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl

Context saved from call 0

state=busy

36

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782

LATCH TREES

ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches

Direct SGA access program output for 9206 instance with too smallshared pool

RDTEX | Latch internals | RUOUG Seminar

ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1

waiter ospid 129 sid 72 pid 45

holding a154b7120 library cache (157) level=5 child=17 kglupc child

waiter ospid 18255 sid 65 pid 930

waiter ospid 6690 sid 554 pid 1654

waiter ospid 4685 sid 879 pid 1034

hellip

waiter ospid 29749 sid 180 pid 155

holding a154b7db8 library cache (157) level=5 child=4 kglupc child

waiter ospid 13104 sid 281 pid 220

waiter ospid 24089 sid 565 pid 636

waiter ospid 25002 sid 621 pid 1481

waiter ospid 16930 sid 1046 pid 783hellip

37

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882

LATCH CLASSES

bull Classes with different spin and wait policies

bull Up to 8 classes By default all the latches belong to class 0

bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2

bull Latch can be assigned to class by its number For example

ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000

ndash _LATCH_CLASSES=46

bull Will be discussed later

RDTEX | Latch internals | RUOUG Seminar

SQLgt select from x$ksllclass

INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4

---- ------ ----- -------- ------ ------ ------ ------ ------

0 20000 0 1 8000 8000 8000 8000 8000

hellip

6 100 12 1000 2000 3000 4000 4000 4000

38

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982

LATCH ACQUISITION IN NO-WAIT MODE

bull kslgetl(laddr0whywhere) ndash Exclusive latch get

bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep

ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches

ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches

bull IMMEDIATE_ statistics are updated

bull Used when many candidate latches are available to bypass the

latch level rule or as part of CBC chain examination

RDTEX | Latch internals | RUOUG Seminar

sqlplus nolog demo_03sql39

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082

LATCH WAITS

RDTEX | Latch internals | RUOUG Seminar 40

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182

WAITING FOR THE LATCH

RDTEX | Latch internals | RUOUG Seminar

983123 G A

C983120983125 1 C983120983125 2

983120983154983151983139983141983155983155 A

983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)

983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144

983116983137983156983139983144

983120983154983151983139983141983155983155 B

41

RDTEX | L h i l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282

LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION

bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries

again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of

times

bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep

bull A sleep corresponds to a wait for the latch free wait event

bull Initially a process sleeps for 001 s This time is doubled in every

subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip

RDTEX | Latch internals | RUOUG Seminar

Still did not found any versionwhich used this

May be it was lt= Oracle 6

42

RDTEX | L t h i t l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382

LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81

Latch gets with wait (kslgetl(laddress1hellip))

ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times

ndash Sleep on latch free event with exponential backoff

ndash Repeat

RDTEX | Latch internals | RUOUG Seminar

43

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482

8I LATCH SPIN CODE FLOW USING DTRACE

RDTEX | Latch internals | RUOUG Seminar

bull hellip Event 10046 trace

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2

983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29

983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270

sqlplus nolog demo_04_8sql 29

44

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582

POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF

bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec

RDTEX | Latch internals | RUOUG Seminar

12 ]2 )1[(

minus= +wait N

timeout

45

bull Most waits were for nothing ndash latch already was free

bull Latch utilization can not be high

bull Unnecessary spins provoke CPU thrashing

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682

92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE

sqlplus nolog demo_05sql 45

RDTEX | Latch internals | RUOUG Seminar

kslgetl(0x50006318 1)

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt kslges(0x50006318 ) -wait latch get

-gt skgslsgts(0x50006318 ) -spin latch get

-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT

-gt kskthbwt(0x0)

-gt kslwlmod() - set up Wait List

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt skgpwwait -sleep latch getsemop(11 17-10 1)

983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140

46

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782

RELIABLE LATCH WAITS

bull Hidden latch wait revolution that we missed Latch relies on waitposting

bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout

bull Exclusive latch spins 20000 cycles by default

bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock

bull Latches assigned to non-default class wait until timeout

bull Version 90 was transient and used mix of backoff and wait posting

This is why _LATCH_WAIT_POSTING parameter disappeared in9i

bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin

RDTEX | Latch internals | RUOUG Seminar

47

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882

IMAGINE THE TIME MICROSCOPE

| a a s | a

Reality One million zoomed time

1 us 1 sec

1 ms 17 min1 sec 115 days

Light speed (300000 kmsec) Sonic speed (300 msec)

CPU tick (2 GHz) 00005 sec

Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec

Max spin time for shared latch (~220005 ticks) ~10 sec

Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec

Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec

OS context switch time (10us-1ms) 10sec-17min

48

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982

LATCH WAITS UNDER TIME MICROSCOPE

Old 8i latch algorithmSpin for latch during 5 seconds

Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min

hellipSpin again for 5 seconds

Sleep again for 23 days (2 sec) hellip

New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)

If spin not succeeded go to sleep until post

According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)

| |

49

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082

_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES

bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and

_LATCH_CLASS_0 parameters

bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000

bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter

bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch

50

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182

TIMED LATCH WAIT POSTING

bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel

bull This can lead to instance hang ndash process will never be woken up

bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout

bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem

bull Generic AIX ports always use timed latch wait posting

bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux

51

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282

FINE GRAIN LATCH TUNING

bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting

bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7

bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo

microseconds using pollsys() (not semtimedop()) system call

ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times

Loop up to ldquoSpinsrdquo cycles

Yield CPU using yield() (or sched_yield())

ndash Sleep for ldquoSleepXrdquo usecs

ndash Then spin again hellip

52

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382

TRACES OF NONDEFAULT CLASS LATCH

983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152

991270

9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000

991270 E983158983141983150983156 10046 983156983154983137983139983141

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482

PERFORMANCE OF LATCH CLASSES

Row cache objects latch contention testcase on ODA (24 SMT)54

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582

IS THE _SPIN_COUNT REALLY STATIC

bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch

waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used

ldquocache buffers chainrdquo latch contention

bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass

(_LATCH_CLASS_X) By default ndash 20000

bull Shared latch spin can be tuned by _SPIN_COUNT value By

default ndash 2000 However this is not trivial

55

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682

SHARED LATCH BEHAVE LIKE ENQUEUE

bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80

bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests

bull This blocking achieved by special 0x40000000 bit in the latch

value This bit is a flag that some incompatible latch acquisitionis in progress

bull Queued processes will be woken up one by one on latch release

bull X mode latch gets effectively serialize the shared latch

bull See my blog for description of experiments

56

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782

TRACE OF SHARED LATCH WAIT

Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the

latch wait list and then spins again

983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141

983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156

983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983147983155983148983159983148983149983151983140()

983147983155983148983155983147983143983155(098316050009BAC )

983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140

983155983141983149983151983152(27)

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882

SHARED LATCH ACQUISITION

bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned

bull X mode shared latch get spins by default up to 4000 cycles

bull S mode does not spin at all (or spins in unknown way)

S mode get X mode get

Held in S mode Compatible 2_SPIN_COUNT

Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT

58

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982

LATCH RELEASE

bull Free the latch ndash kslfre(laddr)

bull Oracle process releases the latch nonatomically

bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process

bull This requires less bus invalidation and ensures propagation of latch

release to other local cachesbull Not fair policy Spinners on the local CPU board have the

preference

bull Then process posts the first process in the list of waiters

59

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082

LATCH CONTENTION

RDTEX | Latch internals | RUOUG Seminar60

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182

LATCH CONTENTION

bull Occurs when a latch is required by several processes at thesame time

bull Mostly due to bugs or abnormal application behavior

bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp

bull To diagnose the latch contention we need to investigate thelatch statistics [8]

61

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282

DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS

Latch requests arrival rate

Immediate gets efficiency

Latch sleeps ratio

Latch wait time per second

Latch spin efficiencymisses

getsspin∆

∆= _σ

misses

sleepsK

∆=

time

gets

∆=λ

gets

misses

∆= ρ

time

timewait W

∆=

_

Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics

62

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382

SAMPLED AVERAGES

By sampling of x$ksupr we can estimate

ndash Length of latch wait queue

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS

ndash Number of spinning processes in 10g

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS

Sampling of v$lathholder estimates latch utilization

Using DTrace we can measure

ndash Latch acquisition and holding times

ndash Actual spin count

w L

s N

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482

DERIVED LATCH STATISTICS

Latch utilization (PASTA)

Average holding time

Length of latch wait list

Recurrent sleeps ratio

Latch acquisition time

time

timeholdinglatchU

∆=asymp

__ ρ

Re_100

___

questsGet

TimeSnap MissGet Pct S

lowast==

λ

W L =

κ

κ σ 1minus+

)(1

W N T saq += minusλ

64

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 25: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2582

THE LATCH STRUCTURE ndash KSLLT

RDTEX | Latch internals | RUOUG Seminar

struct ksllt

ltLatchgt

ldquowhererdquo and ldquowhyrdquo

Level latch class other attributes

Statistics

Latch wait list header

hellip

25

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2682

LATCH SIZE BY VERSION

nix 32bit nix 64bit Windows 32bit

734 92 - 120

806 104 - 104817 104 144 104

901 200 160

920 196 240 200

1010 256 208

1020 - 11203 100 160 104

RDTEX | Latch internals | RUOUG Seminar

x$ksmfsv ndash list of all fixed SGA variables983123E983116EC983124 DI983123983124I983118C983124 983147983155983149983142983155983155983145983162

F983122983119983117 983160$983147983155983149983142983155983158

983127HE983122E 983147983155983149983142983155983156983161983152 = 983147983155983148983148983156

Latch structure was bigger in 101 due to additional latch statistics

26

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2782

ORACLE LATCH IS NOT JUST A SINGLE MEMORY LOCATION

bull Before 11g Value of first latch byte (word for shared latches) wasused to determine latch state

0x00 ndash latch is free

0xFF ndash exclusive latch is busy Was 0x01 in Oracle 7

0x010x02hellip - shared latch holding by 12 hellip processes simultaneously

0x20000000 | pid - shared latch holding exclusively

bull In 11g first latch word shows the pid of the latch holder

0x00 ndash latch is free0x12 ndash Oracle process with pid 18 holds the exclusive latch

RDTEX | Latch internals | RUOUG Seminar

27

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2882

BUSY ORACLE 11G LATCH IN MEMORY

bull 112 Exclusive latch (32-bit platform)

bull 112 Shared latch EXCLUSIVE mode

RDTEX | Latch internals | RUOUG Seminar

SQLgt oradebug peek 200222A0 24

[200222A0 200222B8) = 00000016 00000001 000001D0 00000007

pid^ gets latch level

SQLgt oradebug peek 0xampladdr 24

[2000C814 2000C82C) = 2000001F 0000007C 0000007E 00000006 pid^ gets latch level

983139983151983150983156 983155983153983148983152983148983157983155 983150983151983148983151983143 9831409831419831499831519831350128

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2982

LATCH FIXED TABLES

bull Before Oracle 11g all the latches were externalized to SQL inv$latch (x$ksllt)

bull In 11g parent (fixed SGA) latches are externalized inv$latch_parent (x$kslltr_parent)

bull In 11g child (allocated on startup) latches are externalized in

v$latch_children (x$kslltr_children) bull Oracle 11g v$latch (x$kslltr) merges these data Beware select

from x$kslltr acquires all parent latches and their children

bull Only part of latch structure is visible in x$ksllthellip fixed tables ndash kslltwhr kslltwhy ndash ldquowhererdquo and ldquowhyrdquo latch was acquired last time

ndash kslltwgt hellip - statistics

ndash kslltlvl hellip - attributes

ndash kslltwtt ndash latch wait time

RDTEX | Latch internals | RUOUG Seminar

29

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3082

CUMULATIVE V$LATCH STATISTIC COUNTERS

RDTEX | Latch internals | RUOUG Seminar

Statistics x$ksllt When and how it increments

GETS kslltwgt ldquo++rdquo after wait mode latch get

MISSES kslltwff ldquo++rdquo after wait get if it was missed

SLEEPS kslltwsl ldquo+number_of_sleepsrdquo during get

SPIN_GETS ksllthst0 ldquo++rdquo if get was missed but not slept

WAIT_TIME kslltwtt ldquo+wait_timerdquo after latch get

IMMEDIATE_GETS kslltngt ldquo++rdquo after nowait mode latch get Is notincremented on miss

IMMEDIATE_MISSES kslltnfa ldquo++rdquo if nowait mode get was missed

30

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3182

MANY STATISTICS WERE LOST IN 102

In 102 Oracle made obsolete many useful latch statistics

Latch size was decreased and the following fields always contain 0now in v$latch

ndash v$latchwaiters_woken

ndash v$latchwaits_holding_latch

ndash v$latchsleep1hellipsleep10

ndash and many undocumented statistics

This was mentioned in 102 Reference

RDTEX | Latch internals | RUOUG Seminar

31

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3282

X$KSLLD - LATCH DESCRIPTORS

bull Flags and attributes describing each latch are stored in kslldt arrayof kslld structures

bull [K]ernel [S]ervice [L]atch [L]ock [D]escriptor

bull Some but not all of kslld fields are externalized via x$kslld andv$latchname

bull Located in the PGA

RDTEX | Latch internals | RUOUG Seminar

32

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382

LATCH ATTRIBUTES

bull Each latch have at least the following attributes in kslldt

ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared

ndash PAR Is the latch Solitary or Parent for the family of child latches

ndash G2C Can two child latches be simultaneously requested in wait mode

using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92

ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above

ndash Level 0-14 Used to prevent latch deadlocks

ndash Class 0-7 Spin and wait class assigned to the latch 92 and above

RDTEX | Latch internals | RUOUG Seminar

33

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482

LATCH ATTRIBUTES BY ORACLE VERSION

RDTEX | Latch internals | RUOUG Seminar

11203 553 154 72 - 6 93

Oracle Number of latches PAR G2C LNG UFS SHARED

7340 53 14 2 3 - -

8063 80 21 7 3 - 38174 152 48 19 4 - 9

9208 242 79 37 - - 19

10202 385 114 55 - 4 47

10203 388 117 58 - 4 48

10204 394 117 59 - 4 50

11106 496 145 67 - 6 81

11107 502 145 67 - 6 83

11201 535 149 70 - 6 86

11202 551 153 72 - 6 91

34

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582

LATCH LEVELS

bull Every latch has a level in the range 0-14 to avoid deadlocks

bull Process can request a latch X in wait mode after obtaining latch Y if and

only if level X gt level Y

bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y

bull If these rules are broken the Oracle process raises

ORA-600[504] - trying to obtain another latch at incompatible level Note 281041

ORA-600[525] - trying to obtain another latch at the same level Note 1388881

ORA-600[526] - latch child order violation Note 1388891

RDTEX | Latch internals | RUOUG Seminar

35

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682

PARENT AND CHILD LATCHES

bull Latch can be a parent child or solitary latch

bull Both parent and child latches share the same latch name

bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss

bull Child latches are allocated during instance startup

bull The parent latch can be gotten independently

bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s

RDTEX | Latch internals | RUOUG Seminar

(latch info) wait_event=0 bits=10

holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl

Context saved from call 0

state=busy

36

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782

LATCH TREES

ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches

Direct SGA access program output for 9206 instance with too smallshared pool

RDTEX | Latch internals | RUOUG Seminar

ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1

waiter ospid 129 sid 72 pid 45

holding a154b7120 library cache (157) level=5 child=17 kglupc child

waiter ospid 18255 sid 65 pid 930

waiter ospid 6690 sid 554 pid 1654

waiter ospid 4685 sid 879 pid 1034

hellip

waiter ospid 29749 sid 180 pid 155

holding a154b7db8 library cache (157) level=5 child=4 kglupc child

waiter ospid 13104 sid 281 pid 220

waiter ospid 24089 sid 565 pid 636

waiter ospid 25002 sid 621 pid 1481

waiter ospid 16930 sid 1046 pid 783hellip

37

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882

LATCH CLASSES

bull Classes with different spin and wait policies

bull Up to 8 classes By default all the latches belong to class 0

bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2

bull Latch can be assigned to class by its number For example

ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000

ndash _LATCH_CLASSES=46

bull Will be discussed later

RDTEX | Latch internals | RUOUG Seminar

SQLgt select from x$ksllclass

INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4

---- ------ ----- -------- ------ ------ ------ ------ ------

0 20000 0 1 8000 8000 8000 8000 8000

hellip

6 100 12 1000 2000 3000 4000 4000 4000

38

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982

LATCH ACQUISITION IN NO-WAIT MODE

bull kslgetl(laddr0whywhere) ndash Exclusive latch get

bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep

ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches

ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches

bull IMMEDIATE_ statistics are updated

bull Used when many candidate latches are available to bypass the

latch level rule or as part of CBC chain examination

RDTEX | Latch internals | RUOUG Seminar

sqlplus nolog demo_03sql39

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082

LATCH WAITS

RDTEX | Latch internals | RUOUG Seminar 40

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182

WAITING FOR THE LATCH

RDTEX | Latch internals | RUOUG Seminar

983123 G A

C983120983125 1 C983120983125 2

983120983154983151983139983141983155983155 A

983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)

983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144

983116983137983156983139983144

983120983154983151983139983141983155983155 B

41

RDTEX | L h i l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282

LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION

bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries

again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of

times

bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep

bull A sleep corresponds to a wait for the latch free wait event

bull Initially a process sleeps for 001 s This time is doubled in every

subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip

RDTEX | Latch internals | RUOUG Seminar

Still did not found any versionwhich used this

May be it was lt= Oracle 6

42

RDTEX | L t h i t l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382

LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81

Latch gets with wait (kslgetl(laddress1hellip))

ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times

ndash Sleep on latch free event with exponential backoff

ndash Repeat

RDTEX | Latch internals | RUOUG Seminar

43

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482

8I LATCH SPIN CODE FLOW USING DTRACE

RDTEX | Latch internals | RUOUG Seminar

bull hellip Event 10046 trace

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2

983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29

983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270

sqlplus nolog demo_04_8sql 29

44

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582

POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF

bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec

RDTEX | Latch internals | RUOUG Seminar

12 ]2 )1[(

minus= +wait N

timeout

45

bull Most waits were for nothing ndash latch already was free

bull Latch utilization can not be high

bull Unnecessary spins provoke CPU thrashing

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682

92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE

sqlplus nolog demo_05sql 45

RDTEX | Latch internals | RUOUG Seminar

kslgetl(0x50006318 1)

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt kslges(0x50006318 ) -wait latch get

-gt skgslsgts(0x50006318 ) -spin latch get

-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT

-gt kskthbwt(0x0)

-gt kslwlmod() - set up Wait List

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt skgpwwait -sleep latch getsemop(11 17-10 1)

983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140

46

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782

RELIABLE LATCH WAITS

bull Hidden latch wait revolution that we missed Latch relies on waitposting

bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout

bull Exclusive latch spins 20000 cycles by default

bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock

bull Latches assigned to non-default class wait until timeout

bull Version 90 was transient and used mix of backoff and wait posting

This is why _LATCH_WAIT_POSTING parameter disappeared in9i

bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin

RDTEX | Latch internals | RUOUG Seminar

47

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882

IMAGINE THE TIME MICROSCOPE

| a a s | a

Reality One million zoomed time

1 us 1 sec

1 ms 17 min1 sec 115 days

Light speed (300000 kmsec) Sonic speed (300 msec)

CPU tick (2 GHz) 00005 sec

Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec

Max spin time for shared latch (~220005 ticks) ~10 sec

Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec

Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec

OS context switch time (10us-1ms) 10sec-17min

48

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982

LATCH WAITS UNDER TIME MICROSCOPE

Old 8i latch algorithmSpin for latch during 5 seconds

Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min

hellipSpin again for 5 seconds

Sleep again for 23 days (2 sec) hellip

New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)

If spin not succeeded go to sleep until post

According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)

| |

49

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082

_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES

bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and

_LATCH_CLASS_0 parameters

bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000

bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter

bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch

50

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182

TIMED LATCH WAIT POSTING

bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel

bull This can lead to instance hang ndash process will never be woken up

bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout

bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem

bull Generic AIX ports always use timed latch wait posting

bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux

51

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282

FINE GRAIN LATCH TUNING

bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting

bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7

bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo

microseconds using pollsys() (not semtimedop()) system call

ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times

Loop up to ldquoSpinsrdquo cycles

Yield CPU using yield() (or sched_yield())

ndash Sleep for ldquoSleepXrdquo usecs

ndash Then spin again hellip

52

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382

TRACES OF NONDEFAULT CLASS LATCH

983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152

991270

9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000

991270 E983158983141983150983156 10046 983156983154983137983139983141

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482

PERFORMANCE OF LATCH CLASSES

Row cache objects latch contention testcase on ODA (24 SMT)54

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582

IS THE _SPIN_COUNT REALLY STATIC

bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch

waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used

ldquocache buffers chainrdquo latch contention

bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass

(_LATCH_CLASS_X) By default ndash 20000

bull Shared latch spin can be tuned by _SPIN_COUNT value By

default ndash 2000 However this is not trivial

55

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682

SHARED LATCH BEHAVE LIKE ENQUEUE

bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80

bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests

bull This blocking achieved by special 0x40000000 bit in the latch

value This bit is a flag that some incompatible latch acquisitionis in progress

bull Queued processes will be woken up one by one on latch release

bull X mode latch gets effectively serialize the shared latch

bull See my blog for description of experiments

56

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782

TRACE OF SHARED LATCH WAIT

Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the

latch wait list and then spins again

983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141

983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156

983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983147983155983148983159983148983149983151983140()

983147983155983148983155983147983143983155(098316050009BAC )

983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140

983155983141983149983151983152(27)

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882

SHARED LATCH ACQUISITION

bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned

bull X mode shared latch get spins by default up to 4000 cycles

bull S mode does not spin at all (or spins in unknown way)

S mode get X mode get

Held in S mode Compatible 2_SPIN_COUNT

Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT

58

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982

LATCH RELEASE

bull Free the latch ndash kslfre(laddr)

bull Oracle process releases the latch nonatomically

bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process

bull This requires less bus invalidation and ensures propagation of latch

release to other local cachesbull Not fair policy Spinners on the local CPU board have the

preference

bull Then process posts the first process in the list of waiters

59

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082

LATCH CONTENTION

RDTEX | Latch internals | RUOUG Seminar60

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182

LATCH CONTENTION

bull Occurs when a latch is required by several processes at thesame time

bull Mostly due to bugs or abnormal application behavior

bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp

bull To diagnose the latch contention we need to investigate thelatch statistics [8]

61

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282

DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS

Latch requests arrival rate

Immediate gets efficiency

Latch sleeps ratio

Latch wait time per second

Latch spin efficiencymisses

getsspin∆

∆= _σ

misses

sleepsK

∆=

time

gets

∆=λ

gets

misses

∆= ρ

time

timewait W

∆=

_

Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics

62

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382

SAMPLED AVERAGES

By sampling of x$ksupr we can estimate

ndash Length of latch wait queue

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS

ndash Number of spinning processes in 10g

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS

Sampling of v$lathholder estimates latch utilization

Using DTrace we can measure

ndash Latch acquisition and holding times

ndash Actual spin count

w L

s N

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482

DERIVED LATCH STATISTICS

Latch utilization (PASTA)

Average holding time

Length of latch wait list

Recurrent sleeps ratio

Latch acquisition time

time

timeholdinglatchU

∆=asymp

__ ρ

Re_100

___

questsGet

TimeSnap MissGet Pct S

lowast==

λ

W L =

κ

κ σ 1minus+

)(1

W N T saq += minusλ

64

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 26: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2682

LATCH SIZE BY VERSION

nix 32bit nix 64bit Windows 32bit

734 92 - 120

806 104 - 104817 104 144 104

901 200 160

920 196 240 200

1010 256 208

1020 - 11203 100 160 104

RDTEX | Latch internals | RUOUG Seminar

x$ksmfsv ndash list of all fixed SGA variables983123E983116EC983124 DI983123983124I983118C983124 983147983155983149983142983155983155983145983162

F983122983119983117 983160$983147983155983149983142983155983158

983127HE983122E 983147983155983149983142983155983156983161983152 = 983147983155983148983148983156

Latch structure was bigger in 101 due to additional latch statistics

26

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2782

ORACLE LATCH IS NOT JUST A SINGLE MEMORY LOCATION

bull Before 11g Value of first latch byte (word for shared latches) wasused to determine latch state

0x00 ndash latch is free

0xFF ndash exclusive latch is busy Was 0x01 in Oracle 7

0x010x02hellip - shared latch holding by 12 hellip processes simultaneously

0x20000000 | pid - shared latch holding exclusively

bull In 11g first latch word shows the pid of the latch holder

0x00 ndash latch is free0x12 ndash Oracle process with pid 18 holds the exclusive latch

RDTEX | Latch internals | RUOUG Seminar

27

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2882

BUSY ORACLE 11G LATCH IN MEMORY

bull 112 Exclusive latch (32-bit platform)

bull 112 Shared latch EXCLUSIVE mode

RDTEX | Latch internals | RUOUG Seminar

SQLgt oradebug peek 200222A0 24

[200222A0 200222B8) = 00000016 00000001 000001D0 00000007

pid^ gets latch level

SQLgt oradebug peek 0xampladdr 24

[2000C814 2000C82C) = 2000001F 0000007C 0000007E 00000006 pid^ gets latch level

983139983151983150983156 983155983153983148983152983148983157983155 983150983151983148983151983143 9831409831419831499831519831350128

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2982

LATCH FIXED TABLES

bull Before Oracle 11g all the latches were externalized to SQL inv$latch (x$ksllt)

bull In 11g parent (fixed SGA) latches are externalized inv$latch_parent (x$kslltr_parent)

bull In 11g child (allocated on startup) latches are externalized in

v$latch_children (x$kslltr_children) bull Oracle 11g v$latch (x$kslltr) merges these data Beware select

from x$kslltr acquires all parent latches and their children

bull Only part of latch structure is visible in x$ksllthellip fixed tables ndash kslltwhr kslltwhy ndash ldquowhererdquo and ldquowhyrdquo latch was acquired last time

ndash kslltwgt hellip - statistics

ndash kslltlvl hellip - attributes

ndash kslltwtt ndash latch wait time

RDTEX | Latch internals | RUOUG Seminar

29

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3082

CUMULATIVE V$LATCH STATISTIC COUNTERS

RDTEX | Latch internals | RUOUG Seminar

Statistics x$ksllt When and how it increments

GETS kslltwgt ldquo++rdquo after wait mode latch get

MISSES kslltwff ldquo++rdquo after wait get if it was missed

SLEEPS kslltwsl ldquo+number_of_sleepsrdquo during get

SPIN_GETS ksllthst0 ldquo++rdquo if get was missed but not slept

WAIT_TIME kslltwtt ldquo+wait_timerdquo after latch get

IMMEDIATE_GETS kslltngt ldquo++rdquo after nowait mode latch get Is notincremented on miss

IMMEDIATE_MISSES kslltnfa ldquo++rdquo if nowait mode get was missed

30

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3182

MANY STATISTICS WERE LOST IN 102

In 102 Oracle made obsolete many useful latch statistics

Latch size was decreased and the following fields always contain 0now in v$latch

ndash v$latchwaiters_woken

ndash v$latchwaits_holding_latch

ndash v$latchsleep1hellipsleep10

ndash and many undocumented statistics

This was mentioned in 102 Reference

RDTEX | Latch internals | RUOUG Seminar

31

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3282

X$KSLLD - LATCH DESCRIPTORS

bull Flags and attributes describing each latch are stored in kslldt arrayof kslld structures

bull [K]ernel [S]ervice [L]atch [L]ock [D]escriptor

bull Some but not all of kslld fields are externalized via x$kslld andv$latchname

bull Located in the PGA

RDTEX | Latch internals | RUOUG Seminar

32

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382

LATCH ATTRIBUTES

bull Each latch have at least the following attributes in kslldt

ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared

ndash PAR Is the latch Solitary or Parent for the family of child latches

ndash G2C Can two child latches be simultaneously requested in wait mode

using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92

ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above

ndash Level 0-14 Used to prevent latch deadlocks

ndash Class 0-7 Spin and wait class assigned to the latch 92 and above

RDTEX | Latch internals | RUOUG Seminar

33

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482

LATCH ATTRIBUTES BY ORACLE VERSION

RDTEX | Latch internals | RUOUG Seminar

11203 553 154 72 - 6 93

Oracle Number of latches PAR G2C LNG UFS SHARED

7340 53 14 2 3 - -

8063 80 21 7 3 - 38174 152 48 19 4 - 9

9208 242 79 37 - - 19

10202 385 114 55 - 4 47

10203 388 117 58 - 4 48

10204 394 117 59 - 4 50

11106 496 145 67 - 6 81

11107 502 145 67 - 6 83

11201 535 149 70 - 6 86

11202 551 153 72 - 6 91

34

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582

LATCH LEVELS

bull Every latch has a level in the range 0-14 to avoid deadlocks

bull Process can request a latch X in wait mode after obtaining latch Y if and

only if level X gt level Y

bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y

bull If these rules are broken the Oracle process raises

ORA-600[504] - trying to obtain another latch at incompatible level Note 281041

ORA-600[525] - trying to obtain another latch at the same level Note 1388881

ORA-600[526] - latch child order violation Note 1388891

RDTEX | Latch internals | RUOUG Seminar

35

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682

PARENT AND CHILD LATCHES

bull Latch can be a parent child or solitary latch

bull Both parent and child latches share the same latch name

bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss

bull Child latches are allocated during instance startup

bull The parent latch can be gotten independently

bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s

RDTEX | Latch internals | RUOUG Seminar

(latch info) wait_event=0 bits=10

holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl

Context saved from call 0

state=busy

36

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782

LATCH TREES

ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches

Direct SGA access program output for 9206 instance with too smallshared pool

RDTEX | Latch internals | RUOUG Seminar

ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1

waiter ospid 129 sid 72 pid 45

holding a154b7120 library cache (157) level=5 child=17 kglupc child

waiter ospid 18255 sid 65 pid 930

waiter ospid 6690 sid 554 pid 1654

waiter ospid 4685 sid 879 pid 1034

hellip

waiter ospid 29749 sid 180 pid 155

holding a154b7db8 library cache (157) level=5 child=4 kglupc child

waiter ospid 13104 sid 281 pid 220

waiter ospid 24089 sid 565 pid 636

waiter ospid 25002 sid 621 pid 1481

waiter ospid 16930 sid 1046 pid 783hellip

37

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882

LATCH CLASSES

bull Classes with different spin and wait policies

bull Up to 8 classes By default all the latches belong to class 0

bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2

bull Latch can be assigned to class by its number For example

ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000

ndash _LATCH_CLASSES=46

bull Will be discussed later

RDTEX | Latch internals | RUOUG Seminar

SQLgt select from x$ksllclass

INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4

---- ------ ----- -------- ------ ------ ------ ------ ------

0 20000 0 1 8000 8000 8000 8000 8000

hellip

6 100 12 1000 2000 3000 4000 4000 4000

38

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982

LATCH ACQUISITION IN NO-WAIT MODE

bull kslgetl(laddr0whywhere) ndash Exclusive latch get

bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep

ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches

ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches

bull IMMEDIATE_ statistics are updated

bull Used when many candidate latches are available to bypass the

latch level rule or as part of CBC chain examination

RDTEX | Latch internals | RUOUG Seminar

sqlplus nolog demo_03sql39

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082

LATCH WAITS

RDTEX | Latch internals | RUOUG Seminar 40

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182

WAITING FOR THE LATCH

RDTEX | Latch internals | RUOUG Seminar

983123 G A

C983120983125 1 C983120983125 2

983120983154983151983139983141983155983155 A

983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)

983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144

983116983137983156983139983144

983120983154983151983139983141983155983155 B

41

RDTEX | L h i l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282

LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION

bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries

again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of

times

bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep

bull A sleep corresponds to a wait for the latch free wait event

bull Initially a process sleeps for 001 s This time is doubled in every

subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip

RDTEX | Latch internals | RUOUG Seminar

Still did not found any versionwhich used this

May be it was lt= Oracle 6

42

RDTEX | L t h i t l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382

LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81

Latch gets with wait (kslgetl(laddress1hellip))

ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times

ndash Sleep on latch free event with exponential backoff

ndash Repeat

RDTEX | Latch internals | RUOUG Seminar

43

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482

8I LATCH SPIN CODE FLOW USING DTRACE

RDTEX | Latch internals | RUOUG Seminar

bull hellip Event 10046 trace

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2

983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29

983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270

sqlplus nolog demo_04_8sql 29

44

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582

POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF

bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec

RDTEX | Latch internals | RUOUG Seminar

12 ]2 )1[(

minus= +wait N

timeout

45

bull Most waits were for nothing ndash latch already was free

bull Latch utilization can not be high

bull Unnecessary spins provoke CPU thrashing

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682

92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE

sqlplus nolog demo_05sql 45

RDTEX | Latch internals | RUOUG Seminar

kslgetl(0x50006318 1)

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt kslges(0x50006318 ) -wait latch get

-gt skgslsgts(0x50006318 ) -spin latch get

-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT

-gt kskthbwt(0x0)

-gt kslwlmod() - set up Wait List

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt skgpwwait -sleep latch getsemop(11 17-10 1)

983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140

46

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782

RELIABLE LATCH WAITS

bull Hidden latch wait revolution that we missed Latch relies on waitposting

bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout

bull Exclusive latch spins 20000 cycles by default

bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock

bull Latches assigned to non-default class wait until timeout

bull Version 90 was transient and used mix of backoff and wait posting

This is why _LATCH_WAIT_POSTING parameter disappeared in9i

bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin

RDTEX | Latch internals | RUOUG Seminar

47

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882

IMAGINE THE TIME MICROSCOPE

| a a s | a

Reality One million zoomed time

1 us 1 sec

1 ms 17 min1 sec 115 days

Light speed (300000 kmsec) Sonic speed (300 msec)

CPU tick (2 GHz) 00005 sec

Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec

Max spin time for shared latch (~220005 ticks) ~10 sec

Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec

Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec

OS context switch time (10us-1ms) 10sec-17min

48

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982

LATCH WAITS UNDER TIME MICROSCOPE

Old 8i latch algorithmSpin for latch during 5 seconds

Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min

hellipSpin again for 5 seconds

Sleep again for 23 days (2 sec) hellip

New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)

If spin not succeeded go to sleep until post

According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)

| |

49

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082

_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES

bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and

_LATCH_CLASS_0 parameters

bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000

bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter

bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch

50

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182

TIMED LATCH WAIT POSTING

bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel

bull This can lead to instance hang ndash process will never be woken up

bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout

bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem

bull Generic AIX ports always use timed latch wait posting

bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux

51

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282

FINE GRAIN LATCH TUNING

bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting

bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7

bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo

microseconds using pollsys() (not semtimedop()) system call

ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times

Loop up to ldquoSpinsrdquo cycles

Yield CPU using yield() (or sched_yield())

ndash Sleep for ldquoSleepXrdquo usecs

ndash Then spin again hellip

52

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382

TRACES OF NONDEFAULT CLASS LATCH

983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152

991270

9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000

991270 E983158983141983150983156 10046 983156983154983137983139983141

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482

PERFORMANCE OF LATCH CLASSES

Row cache objects latch contention testcase on ODA (24 SMT)54

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582

IS THE _SPIN_COUNT REALLY STATIC

bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch

waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used

ldquocache buffers chainrdquo latch contention

bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass

(_LATCH_CLASS_X) By default ndash 20000

bull Shared latch spin can be tuned by _SPIN_COUNT value By

default ndash 2000 However this is not trivial

55

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682

SHARED LATCH BEHAVE LIKE ENQUEUE

bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80

bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests

bull This blocking achieved by special 0x40000000 bit in the latch

value This bit is a flag that some incompatible latch acquisitionis in progress

bull Queued processes will be woken up one by one on latch release

bull X mode latch gets effectively serialize the shared latch

bull See my blog for description of experiments

56

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782

TRACE OF SHARED LATCH WAIT

Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the

latch wait list and then spins again

983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141

983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156

983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983147983155983148983159983148983149983151983140()

983147983155983148983155983147983143983155(098316050009BAC )

983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140

983155983141983149983151983152(27)

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882

SHARED LATCH ACQUISITION

bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned

bull X mode shared latch get spins by default up to 4000 cycles

bull S mode does not spin at all (or spins in unknown way)

S mode get X mode get

Held in S mode Compatible 2_SPIN_COUNT

Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT

58

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982

LATCH RELEASE

bull Free the latch ndash kslfre(laddr)

bull Oracle process releases the latch nonatomically

bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process

bull This requires less bus invalidation and ensures propagation of latch

release to other local cachesbull Not fair policy Spinners on the local CPU board have the

preference

bull Then process posts the first process in the list of waiters

59

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082

LATCH CONTENTION

RDTEX | Latch internals | RUOUG Seminar60

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182

LATCH CONTENTION

bull Occurs when a latch is required by several processes at thesame time

bull Mostly due to bugs or abnormal application behavior

bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp

bull To diagnose the latch contention we need to investigate thelatch statistics [8]

61

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282

DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS

Latch requests arrival rate

Immediate gets efficiency

Latch sleeps ratio

Latch wait time per second

Latch spin efficiencymisses

getsspin∆

∆= _σ

misses

sleepsK

∆=

time

gets

∆=λ

gets

misses

∆= ρ

time

timewait W

∆=

_

Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics

62

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382

SAMPLED AVERAGES

By sampling of x$ksupr we can estimate

ndash Length of latch wait queue

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS

ndash Number of spinning processes in 10g

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS

Sampling of v$lathholder estimates latch utilization

Using DTrace we can measure

ndash Latch acquisition and holding times

ndash Actual spin count

w L

s N

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482

DERIVED LATCH STATISTICS

Latch utilization (PASTA)

Average holding time

Length of latch wait list

Recurrent sleeps ratio

Latch acquisition time

time

timeholdinglatchU

∆=asymp

__ ρ

Re_100

___

questsGet

TimeSnap MissGet Pct S

lowast==

λ

W L =

κ

κ σ 1minus+

)(1

W N T saq += minusλ

64

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 27: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2782

ORACLE LATCH IS NOT JUST A SINGLE MEMORY LOCATION

bull Before 11g Value of first latch byte (word for shared latches) wasused to determine latch state

0x00 ndash latch is free

0xFF ndash exclusive latch is busy Was 0x01 in Oracle 7

0x010x02hellip - shared latch holding by 12 hellip processes simultaneously

0x20000000 | pid - shared latch holding exclusively

bull In 11g first latch word shows the pid of the latch holder

0x00 ndash latch is free0x12 ndash Oracle process with pid 18 holds the exclusive latch

RDTEX | Latch internals | RUOUG Seminar

27

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2882

BUSY ORACLE 11G LATCH IN MEMORY

bull 112 Exclusive latch (32-bit platform)

bull 112 Shared latch EXCLUSIVE mode

RDTEX | Latch internals | RUOUG Seminar

SQLgt oradebug peek 200222A0 24

[200222A0 200222B8) = 00000016 00000001 000001D0 00000007

pid^ gets latch level

SQLgt oradebug peek 0xampladdr 24

[2000C814 2000C82C) = 2000001F 0000007C 0000007E 00000006 pid^ gets latch level

983139983151983150983156 983155983153983148983152983148983157983155 983150983151983148983151983143 9831409831419831499831519831350128

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2982

LATCH FIXED TABLES

bull Before Oracle 11g all the latches were externalized to SQL inv$latch (x$ksllt)

bull In 11g parent (fixed SGA) latches are externalized inv$latch_parent (x$kslltr_parent)

bull In 11g child (allocated on startup) latches are externalized in

v$latch_children (x$kslltr_children) bull Oracle 11g v$latch (x$kslltr) merges these data Beware select

from x$kslltr acquires all parent latches and their children

bull Only part of latch structure is visible in x$ksllthellip fixed tables ndash kslltwhr kslltwhy ndash ldquowhererdquo and ldquowhyrdquo latch was acquired last time

ndash kslltwgt hellip - statistics

ndash kslltlvl hellip - attributes

ndash kslltwtt ndash latch wait time

RDTEX | Latch internals | RUOUG Seminar

29

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3082

CUMULATIVE V$LATCH STATISTIC COUNTERS

RDTEX | Latch internals | RUOUG Seminar

Statistics x$ksllt When and how it increments

GETS kslltwgt ldquo++rdquo after wait mode latch get

MISSES kslltwff ldquo++rdquo after wait get if it was missed

SLEEPS kslltwsl ldquo+number_of_sleepsrdquo during get

SPIN_GETS ksllthst0 ldquo++rdquo if get was missed but not slept

WAIT_TIME kslltwtt ldquo+wait_timerdquo after latch get

IMMEDIATE_GETS kslltngt ldquo++rdquo after nowait mode latch get Is notincremented on miss

IMMEDIATE_MISSES kslltnfa ldquo++rdquo if nowait mode get was missed

30

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3182

MANY STATISTICS WERE LOST IN 102

In 102 Oracle made obsolete many useful latch statistics

Latch size was decreased and the following fields always contain 0now in v$latch

ndash v$latchwaiters_woken

ndash v$latchwaits_holding_latch

ndash v$latchsleep1hellipsleep10

ndash and many undocumented statistics

This was mentioned in 102 Reference

RDTEX | Latch internals | RUOUG Seminar

31

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3282

X$KSLLD - LATCH DESCRIPTORS

bull Flags and attributes describing each latch are stored in kslldt arrayof kslld structures

bull [K]ernel [S]ervice [L]atch [L]ock [D]escriptor

bull Some but not all of kslld fields are externalized via x$kslld andv$latchname

bull Located in the PGA

RDTEX | Latch internals | RUOUG Seminar

32

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382

LATCH ATTRIBUTES

bull Each latch have at least the following attributes in kslldt

ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared

ndash PAR Is the latch Solitary or Parent for the family of child latches

ndash G2C Can two child latches be simultaneously requested in wait mode

using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92

ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above

ndash Level 0-14 Used to prevent latch deadlocks

ndash Class 0-7 Spin and wait class assigned to the latch 92 and above

RDTEX | Latch internals | RUOUG Seminar

33

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482

LATCH ATTRIBUTES BY ORACLE VERSION

RDTEX | Latch internals | RUOUG Seminar

11203 553 154 72 - 6 93

Oracle Number of latches PAR G2C LNG UFS SHARED

7340 53 14 2 3 - -

8063 80 21 7 3 - 38174 152 48 19 4 - 9

9208 242 79 37 - - 19

10202 385 114 55 - 4 47

10203 388 117 58 - 4 48

10204 394 117 59 - 4 50

11106 496 145 67 - 6 81

11107 502 145 67 - 6 83

11201 535 149 70 - 6 86

11202 551 153 72 - 6 91

34

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582

LATCH LEVELS

bull Every latch has a level in the range 0-14 to avoid deadlocks

bull Process can request a latch X in wait mode after obtaining latch Y if and

only if level X gt level Y

bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y

bull If these rules are broken the Oracle process raises

ORA-600[504] - trying to obtain another latch at incompatible level Note 281041

ORA-600[525] - trying to obtain another latch at the same level Note 1388881

ORA-600[526] - latch child order violation Note 1388891

RDTEX | Latch internals | RUOUG Seminar

35

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682

PARENT AND CHILD LATCHES

bull Latch can be a parent child or solitary latch

bull Both parent and child latches share the same latch name

bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss

bull Child latches are allocated during instance startup

bull The parent latch can be gotten independently

bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s

RDTEX | Latch internals | RUOUG Seminar

(latch info) wait_event=0 bits=10

holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl

Context saved from call 0

state=busy

36

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782

LATCH TREES

ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches

Direct SGA access program output for 9206 instance with too smallshared pool

RDTEX | Latch internals | RUOUG Seminar

ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1

waiter ospid 129 sid 72 pid 45

holding a154b7120 library cache (157) level=5 child=17 kglupc child

waiter ospid 18255 sid 65 pid 930

waiter ospid 6690 sid 554 pid 1654

waiter ospid 4685 sid 879 pid 1034

hellip

waiter ospid 29749 sid 180 pid 155

holding a154b7db8 library cache (157) level=5 child=4 kglupc child

waiter ospid 13104 sid 281 pid 220

waiter ospid 24089 sid 565 pid 636

waiter ospid 25002 sid 621 pid 1481

waiter ospid 16930 sid 1046 pid 783hellip

37

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882

LATCH CLASSES

bull Classes with different spin and wait policies

bull Up to 8 classes By default all the latches belong to class 0

bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2

bull Latch can be assigned to class by its number For example

ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000

ndash _LATCH_CLASSES=46

bull Will be discussed later

RDTEX | Latch internals | RUOUG Seminar

SQLgt select from x$ksllclass

INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4

---- ------ ----- -------- ------ ------ ------ ------ ------

0 20000 0 1 8000 8000 8000 8000 8000

hellip

6 100 12 1000 2000 3000 4000 4000 4000

38

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982

LATCH ACQUISITION IN NO-WAIT MODE

bull kslgetl(laddr0whywhere) ndash Exclusive latch get

bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep

ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches

ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches

bull IMMEDIATE_ statistics are updated

bull Used when many candidate latches are available to bypass the

latch level rule or as part of CBC chain examination

RDTEX | Latch internals | RUOUG Seminar

sqlplus nolog demo_03sql39

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082

LATCH WAITS

RDTEX | Latch internals | RUOUG Seminar 40

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182

WAITING FOR THE LATCH

RDTEX | Latch internals | RUOUG Seminar

983123 G A

C983120983125 1 C983120983125 2

983120983154983151983139983141983155983155 A

983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)

983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144

983116983137983156983139983144

983120983154983151983139983141983155983155 B

41

RDTEX | L h i l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282

LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION

bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries

again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of

times

bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep

bull A sleep corresponds to a wait for the latch free wait event

bull Initially a process sleeps for 001 s This time is doubled in every

subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip

RDTEX | Latch internals | RUOUG Seminar

Still did not found any versionwhich used this

May be it was lt= Oracle 6

42

RDTEX | L t h i t l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382

LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81

Latch gets with wait (kslgetl(laddress1hellip))

ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times

ndash Sleep on latch free event with exponential backoff

ndash Repeat

RDTEX | Latch internals | RUOUG Seminar

43

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482

8I LATCH SPIN CODE FLOW USING DTRACE

RDTEX | Latch internals | RUOUG Seminar

bull hellip Event 10046 trace

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2

983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29

983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270

sqlplus nolog demo_04_8sql 29

44

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582

POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF

bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec

RDTEX | Latch internals | RUOUG Seminar

12 ]2 )1[(

minus= +wait N

timeout

45

bull Most waits were for nothing ndash latch already was free

bull Latch utilization can not be high

bull Unnecessary spins provoke CPU thrashing

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682

92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE

sqlplus nolog demo_05sql 45

RDTEX | Latch internals | RUOUG Seminar

kslgetl(0x50006318 1)

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt kslges(0x50006318 ) -wait latch get

-gt skgslsgts(0x50006318 ) -spin latch get

-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT

-gt kskthbwt(0x0)

-gt kslwlmod() - set up Wait List

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt skgpwwait -sleep latch getsemop(11 17-10 1)

983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140

46

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782

RELIABLE LATCH WAITS

bull Hidden latch wait revolution that we missed Latch relies on waitposting

bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout

bull Exclusive latch spins 20000 cycles by default

bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock

bull Latches assigned to non-default class wait until timeout

bull Version 90 was transient and used mix of backoff and wait posting

This is why _LATCH_WAIT_POSTING parameter disappeared in9i

bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin

RDTEX | Latch internals | RUOUG Seminar

47

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882

IMAGINE THE TIME MICROSCOPE

| a a s | a

Reality One million zoomed time

1 us 1 sec

1 ms 17 min1 sec 115 days

Light speed (300000 kmsec) Sonic speed (300 msec)

CPU tick (2 GHz) 00005 sec

Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec

Max spin time for shared latch (~220005 ticks) ~10 sec

Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec

Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec

OS context switch time (10us-1ms) 10sec-17min

48

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982

LATCH WAITS UNDER TIME MICROSCOPE

Old 8i latch algorithmSpin for latch during 5 seconds

Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min

hellipSpin again for 5 seconds

Sleep again for 23 days (2 sec) hellip

New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)

If spin not succeeded go to sleep until post

According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)

| |

49

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082

_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES

bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and

_LATCH_CLASS_0 parameters

bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000

bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter

bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch

50

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182

TIMED LATCH WAIT POSTING

bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel

bull This can lead to instance hang ndash process will never be woken up

bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout

bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem

bull Generic AIX ports always use timed latch wait posting

bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux

51

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282

FINE GRAIN LATCH TUNING

bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting

bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7

bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo

microseconds using pollsys() (not semtimedop()) system call

ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times

Loop up to ldquoSpinsrdquo cycles

Yield CPU using yield() (or sched_yield())

ndash Sleep for ldquoSleepXrdquo usecs

ndash Then spin again hellip

52

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382

TRACES OF NONDEFAULT CLASS LATCH

983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152

991270

9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000

991270 E983158983141983150983156 10046 983156983154983137983139983141

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482

PERFORMANCE OF LATCH CLASSES

Row cache objects latch contention testcase on ODA (24 SMT)54

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582

IS THE _SPIN_COUNT REALLY STATIC

bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch

waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used

ldquocache buffers chainrdquo latch contention

bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass

(_LATCH_CLASS_X) By default ndash 20000

bull Shared latch spin can be tuned by _SPIN_COUNT value By

default ndash 2000 However this is not trivial

55

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682

SHARED LATCH BEHAVE LIKE ENQUEUE

bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80

bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests

bull This blocking achieved by special 0x40000000 bit in the latch

value This bit is a flag that some incompatible latch acquisitionis in progress

bull Queued processes will be woken up one by one on latch release

bull X mode latch gets effectively serialize the shared latch

bull See my blog for description of experiments

56

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782

TRACE OF SHARED LATCH WAIT

Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the

latch wait list and then spins again

983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141

983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156

983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983147983155983148983159983148983149983151983140()

983147983155983148983155983147983143983155(098316050009BAC )

983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140

983155983141983149983151983152(27)

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882

SHARED LATCH ACQUISITION

bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned

bull X mode shared latch get spins by default up to 4000 cycles

bull S mode does not spin at all (or spins in unknown way)

S mode get X mode get

Held in S mode Compatible 2_SPIN_COUNT

Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT

58

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982

LATCH RELEASE

bull Free the latch ndash kslfre(laddr)

bull Oracle process releases the latch nonatomically

bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process

bull This requires less bus invalidation and ensures propagation of latch

release to other local cachesbull Not fair policy Spinners on the local CPU board have the

preference

bull Then process posts the first process in the list of waiters

59

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082

LATCH CONTENTION

RDTEX | Latch internals | RUOUG Seminar60

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182

LATCH CONTENTION

bull Occurs when a latch is required by several processes at thesame time

bull Mostly due to bugs or abnormal application behavior

bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp

bull To diagnose the latch contention we need to investigate thelatch statistics [8]

61

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282

DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS

Latch requests arrival rate

Immediate gets efficiency

Latch sleeps ratio

Latch wait time per second

Latch spin efficiencymisses

getsspin∆

∆= _σ

misses

sleepsK

∆=

time

gets

∆=λ

gets

misses

∆= ρ

time

timewait W

∆=

_

Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics

62

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382

SAMPLED AVERAGES

By sampling of x$ksupr we can estimate

ndash Length of latch wait queue

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS

ndash Number of spinning processes in 10g

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS

Sampling of v$lathholder estimates latch utilization

Using DTrace we can measure

ndash Latch acquisition and holding times

ndash Actual spin count

w L

s N

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482

DERIVED LATCH STATISTICS

Latch utilization (PASTA)

Average holding time

Length of latch wait list

Recurrent sleeps ratio

Latch acquisition time

time

timeholdinglatchU

∆=asymp

__ ρ

Re_100

___

questsGet

TimeSnap MissGet Pct S

lowast==

λ

W L =

κ

κ σ 1minus+

)(1

W N T saq += minusλ

64

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 28: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2882

BUSY ORACLE 11G LATCH IN MEMORY

bull 112 Exclusive latch (32-bit platform)

bull 112 Shared latch EXCLUSIVE mode

RDTEX | Latch internals | RUOUG Seminar

SQLgt oradebug peek 200222A0 24

[200222A0 200222B8) = 00000016 00000001 000001D0 00000007

pid^ gets latch level

SQLgt oradebug peek 0xampladdr 24

[2000C814 2000C82C) = 2000001F 0000007C 0000007E 00000006 pid^ gets latch level

983139983151983150983156 983155983153983148983152983148983157983155 983150983151983148983151983143 9831409831419831499831519831350128

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2982

LATCH FIXED TABLES

bull Before Oracle 11g all the latches were externalized to SQL inv$latch (x$ksllt)

bull In 11g parent (fixed SGA) latches are externalized inv$latch_parent (x$kslltr_parent)

bull In 11g child (allocated on startup) latches are externalized in

v$latch_children (x$kslltr_children) bull Oracle 11g v$latch (x$kslltr) merges these data Beware select

from x$kslltr acquires all parent latches and their children

bull Only part of latch structure is visible in x$ksllthellip fixed tables ndash kslltwhr kslltwhy ndash ldquowhererdquo and ldquowhyrdquo latch was acquired last time

ndash kslltwgt hellip - statistics

ndash kslltlvl hellip - attributes

ndash kslltwtt ndash latch wait time

RDTEX | Latch internals | RUOUG Seminar

29

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3082

CUMULATIVE V$LATCH STATISTIC COUNTERS

RDTEX | Latch internals | RUOUG Seminar

Statistics x$ksllt When and how it increments

GETS kslltwgt ldquo++rdquo after wait mode latch get

MISSES kslltwff ldquo++rdquo after wait get if it was missed

SLEEPS kslltwsl ldquo+number_of_sleepsrdquo during get

SPIN_GETS ksllthst0 ldquo++rdquo if get was missed but not slept

WAIT_TIME kslltwtt ldquo+wait_timerdquo after latch get

IMMEDIATE_GETS kslltngt ldquo++rdquo after nowait mode latch get Is notincremented on miss

IMMEDIATE_MISSES kslltnfa ldquo++rdquo if nowait mode get was missed

30

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3182

MANY STATISTICS WERE LOST IN 102

In 102 Oracle made obsolete many useful latch statistics

Latch size was decreased and the following fields always contain 0now in v$latch

ndash v$latchwaiters_woken

ndash v$latchwaits_holding_latch

ndash v$latchsleep1hellipsleep10

ndash and many undocumented statistics

This was mentioned in 102 Reference

RDTEX | Latch internals | RUOUG Seminar

31

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3282

X$KSLLD - LATCH DESCRIPTORS

bull Flags and attributes describing each latch are stored in kslldt arrayof kslld structures

bull [K]ernel [S]ervice [L]atch [L]ock [D]escriptor

bull Some but not all of kslld fields are externalized via x$kslld andv$latchname

bull Located in the PGA

RDTEX | Latch internals | RUOUG Seminar

32

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382

LATCH ATTRIBUTES

bull Each latch have at least the following attributes in kslldt

ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared

ndash PAR Is the latch Solitary or Parent for the family of child latches

ndash G2C Can two child latches be simultaneously requested in wait mode

using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92

ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above

ndash Level 0-14 Used to prevent latch deadlocks

ndash Class 0-7 Spin and wait class assigned to the latch 92 and above

RDTEX | Latch internals | RUOUG Seminar

33

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482

LATCH ATTRIBUTES BY ORACLE VERSION

RDTEX | Latch internals | RUOUG Seminar

11203 553 154 72 - 6 93

Oracle Number of latches PAR G2C LNG UFS SHARED

7340 53 14 2 3 - -

8063 80 21 7 3 - 38174 152 48 19 4 - 9

9208 242 79 37 - - 19

10202 385 114 55 - 4 47

10203 388 117 58 - 4 48

10204 394 117 59 - 4 50

11106 496 145 67 - 6 81

11107 502 145 67 - 6 83

11201 535 149 70 - 6 86

11202 551 153 72 - 6 91

34

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582

LATCH LEVELS

bull Every latch has a level in the range 0-14 to avoid deadlocks

bull Process can request a latch X in wait mode after obtaining latch Y if and

only if level X gt level Y

bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y

bull If these rules are broken the Oracle process raises

ORA-600[504] - trying to obtain another latch at incompatible level Note 281041

ORA-600[525] - trying to obtain another latch at the same level Note 1388881

ORA-600[526] - latch child order violation Note 1388891

RDTEX | Latch internals | RUOUG Seminar

35

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682

PARENT AND CHILD LATCHES

bull Latch can be a parent child or solitary latch

bull Both parent and child latches share the same latch name

bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss

bull Child latches are allocated during instance startup

bull The parent latch can be gotten independently

bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s

RDTEX | Latch internals | RUOUG Seminar

(latch info) wait_event=0 bits=10

holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl

Context saved from call 0

state=busy

36

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782

LATCH TREES

ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches

Direct SGA access program output for 9206 instance with too smallshared pool

RDTEX | Latch internals | RUOUG Seminar

ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1

waiter ospid 129 sid 72 pid 45

holding a154b7120 library cache (157) level=5 child=17 kglupc child

waiter ospid 18255 sid 65 pid 930

waiter ospid 6690 sid 554 pid 1654

waiter ospid 4685 sid 879 pid 1034

hellip

waiter ospid 29749 sid 180 pid 155

holding a154b7db8 library cache (157) level=5 child=4 kglupc child

waiter ospid 13104 sid 281 pid 220

waiter ospid 24089 sid 565 pid 636

waiter ospid 25002 sid 621 pid 1481

waiter ospid 16930 sid 1046 pid 783hellip

37

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882

LATCH CLASSES

bull Classes with different spin and wait policies

bull Up to 8 classes By default all the latches belong to class 0

bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2

bull Latch can be assigned to class by its number For example

ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000

ndash _LATCH_CLASSES=46

bull Will be discussed later

RDTEX | Latch internals | RUOUG Seminar

SQLgt select from x$ksllclass

INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4

---- ------ ----- -------- ------ ------ ------ ------ ------

0 20000 0 1 8000 8000 8000 8000 8000

hellip

6 100 12 1000 2000 3000 4000 4000 4000

38

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982

LATCH ACQUISITION IN NO-WAIT MODE

bull kslgetl(laddr0whywhere) ndash Exclusive latch get

bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep

ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches

ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches

bull IMMEDIATE_ statistics are updated

bull Used when many candidate latches are available to bypass the

latch level rule or as part of CBC chain examination

RDTEX | Latch internals | RUOUG Seminar

sqlplus nolog demo_03sql39

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082

LATCH WAITS

RDTEX | Latch internals | RUOUG Seminar 40

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182

WAITING FOR THE LATCH

RDTEX | Latch internals | RUOUG Seminar

983123 G A

C983120983125 1 C983120983125 2

983120983154983151983139983141983155983155 A

983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)

983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144

983116983137983156983139983144

983120983154983151983139983141983155983155 B

41

RDTEX | L h i l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282

LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION

bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries

again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of

times

bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep

bull A sleep corresponds to a wait for the latch free wait event

bull Initially a process sleeps for 001 s This time is doubled in every

subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip

RDTEX | Latch internals | RUOUG Seminar

Still did not found any versionwhich used this

May be it was lt= Oracle 6

42

RDTEX | L t h i t l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382

LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81

Latch gets with wait (kslgetl(laddress1hellip))

ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times

ndash Sleep on latch free event with exponential backoff

ndash Repeat

RDTEX | Latch internals | RUOUG Seminar

43

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482

8I LATCH SPIN CODE FLOW USING DTRACE

RDTEX | Latch internals | RUOUG Seminar

bull hellip Event 10046 trace

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2

983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29

983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270

sqlplus nolog demo_04_8sql 29

44

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582

POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF

bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec

RDTEX | Latch internals | RUOUG Seminar

12 ]2 )1[(

minus= +wait N

timeout

45

bull Most waits were for nothing ndash latch already was free

bull Latch utilization can not be high

bull Unnecessary spins provoke CPU thrashing

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682

92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE

sqlplus nolog demo_05sql 45

RDTEX | Latch internals | RUOUG Seminar

kslgetl(0x50006318 1)

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt kslges(0x50006318 ) -wait latch get

-gt skgslsgts(0x50006318 ) -spin latch get

-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT

-gt kskthbwt(0x0)

-gt kslwlmod() - set up Wait List

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt skgpwwait -sleep latch getsemop(11 17-10 1)

983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140

46

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782

RELIABLE LATCH WAITS

bull Hidden latch wait revolution that we missed Latch relies on waitposting

bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout

bull Exclusive latch spins 20000 cycles by default

bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock

bull Latches assigned to non-default class wait until timeout

bull Version 90 was transient and used mix of backoff and wait posting

This is why _LATCH_WAIT_POSTING parameter disappeared in9i

bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin

RDTEX | Latch internals | RUOUG Seminar

47

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882

IMAGINE THE TIME MICROSCOPE

| a a s | a

Reality One million zoomed time

1 us 1 sec

1 ms 17 min1 sec 115 days

Light speed (300000 kmsec) Sonic speed (300 msec)

CPU tick (2 GHz) 00005 sec

Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec

Max spin time for shared latch (~220005 ticks) ~10 sec

Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec

Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec

OS context switch time (10us-1ms) 10sec-17min

48

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982

LATCH WAITS UNDER TIME MICROSCOPE

Old 8i latch algorithmSpin for latch during 5 seconds

Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min

hellipSpin again for 5 seconds

Sleep again for 23 days (2 sec) hellip

New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)

If spin not succeeded go to sleep until post

According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)

| |

49

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082

_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES

bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and

_LATCH_CLASS_0 parameters

bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000

bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter

bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch

50

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182

TIMED LATCH WAIT POSTING

bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel

bull This can lead to instance hang ndash process will never be woken up

bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout

bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem

bull Generic AIX ports always use timed latch wait posting

bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux

51

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282

FINE GRAIN LATCH TUNING

bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting

bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7

bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo

microseconds using pollsys() (not semtimedop()) system call

ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times

Loop up to ldquoSpinsrdquo cycles

Yield CPU using yield() (or sched_yield())

ndash Sleep for ldquoSleepXrdquo usecs

ndash Then spin again hellip

52

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382

TRACES OF NONDEFAULT CLASS LATCH

983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152

991270

9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000

991270 E983158983141983150983156 10046 983156983154983137983139983141

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482

PERFORMANCE OF LATCH CLASSES

Row cache objects latch contention testcase on ODA (24 SMT)54

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582

IS THE _SPIN_COUNT REALLY STATIC

bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch

waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used

ldquocache buffers chainrdquo latch contention

bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass

(_LATCH_CLASS_X) By default ndash 20000

bull Shared latch spin can be tuned by _SPIN_COUNT value By

default ndash 2000 However this is not trivial

55

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682

SHARED LATCH BEHAVE LIKE ENQUEUE

bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80

bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests

bull This blocking achieved by special 0x40000000 bit in the latch

value This bit is a flag that some incompatible latch acquisitionis in progress

bull Queued processes will be woken up one by one on latch release

bull X mode latch gets effectively serialize the shared latch

bull See my blog for description of experiments

56

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782

TRACE OF SHARED LATCH WAIT

Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the

latch wait list and then spins again

983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141

983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156

983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983147983155983148983159983148983149983151983140()

983147983155983148983155983147983143983155(098316050009BAC )

983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140

983155983141983149983151983152(27)

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882

SHARED LATCH ACQUISITION

bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned

bull X mode shared latch get spins by default up to 4000 cycles

bull S mode does not spin at all (or spins in unknown way)

S mode get X mode get

Held in S mode Compatible 2_SPIN_COUNT

Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT

58

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982

LATCH RELEASE

bull Free the latch ndash kslfre(laddr)

bull Oracle process releases the latch nonatomically

bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process

bull This requires less bus invalidation and ensures propagation of latch

release to other local cachesbull Not fair policy Spinners on the local CPU board have the

preference

bull Then process posts the first process in the list of waiters

59

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082

LATCH CONTENTION

RDTEX | Latch internals | RUOUG Seminar60

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182

LATCH CONTENTION

bull Occurs when a latch is required by several processes at thesame time

bull Mostly due to bugs or abnormal application behavior

bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp

bull To diagnose the latch contention we need to investigate thelatch statistics [8]

61

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282

DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS

Latch requests arrival rate

Immediate gets efficiency

Latch sleeps ratio

Latch wait time per second

Latch spin efficiencymisses

getsspin∆

∆= _σ

misses

sleepsK

∆=

time

gets

∆=λ

gets

misses

∆= ρ

time

timewait W

∆=

_

Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics

62

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382

SAMPLED AVERAGES

By sampling of x$ksupr we can estimate

ndash Length of latch wait queue

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS

ndash Number of spinning processes in 10g

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS

Sampling of v$lathholder estimates latch utilization

Using DTrace we can measure

ndash Latch acquisition and holding times

ndash Actual spin count

w L

s N

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482

DERIVED LATCH STATISTICS

Latch utilization (PASTA)

Average holding time

Length of latch wait list

Recurrent sleeps ratio

Latch acquisition time

time

timeholdinglatchU

∆=asymp

__ ρ

Re_100

___

questsGet

TimeSnap MissGet Pct S

lowast==

λ

W L =

κ

κ σ 1minus+

)(1

W N T saq += minusλ

64

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 29: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 2982

LATCH FIXED TABLES

bull Before Oracle 11g all the latches were externalized to SQL inv$latch (x$ksllt)

bull In 11g parent (fixed SGA) latches are externalized inv$latch_parent (x$kslltr_parent)

bull In 11g child (allocated on startup) latches are externalized in

v$latch_children (x$kslltr_children) bull Oracle 11g v$latch (x$kslltr) merges these data Beware select

from x$kslltr acquires all parent latches and their children

bull Only part of latch structure is visible in x$ksllthellip fixed tables ndash kslltwhr kslltwhy ndash ldquowhererdquo and ldquowhyrdquo latch was acquired last time

ndash kslltwgt hellip - statistics

ndash kslltlvl hellip - attributes

ndash kslltwtt ndash latch wait time

RDTEX | Latch internals | RUOUG Seminar

29

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3082

CUMULATIVE V$LATCH STATISTIC COUNTERS

RDTEX | Latch internals | RUOUG Seminar

Statistics x$ksllt When and how it increments

GETS kslltwgt ldquo++rdquo after wait mode latch get

MISSES kslltwff ldquo++rdquo after wait get if it was missed

SLEEPS kslltwsl ldquo+number_of_sleepsrdquo during get

SPIN_GETS ksllthst0 ldquo++rdquo if get was missed but not slept

WAIT_TIME kslltwtt ldquo+wait_timerdquo after latch get

IMMEDIATE_GETS kslltngt ldquo++rdquo after nowait mode latch get Is notincremented on miss

IMMEDIATE_MISSES kslltnfa ldquo++rdquo if nowait mode get was missed

30

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3182

MANY STATISTICS WERE LOST IN 102

In 102 Oracle made obsolete many useful latch statistics

Latch size was decreased and the following fields always contain 0now in v$latch

ndash v$latchwaiters_woken

ndash v$latchwaits_holding_latch

ndash v$latchsleep1hellipsleep10

ndash and many undocumented statistics

This was mentioned in 102 Reference

RDTEX | Latch internals | RUOUG Seminar

31

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3282

X$KSLLD - LATCH DESCRIPTORS

bull Flags and attributes describing each latch are stored in kslldt arrayof kslld structures

bull [K]ernel [S]ervice [L]atch [L]ock [D]escriptor

bull Some but not all of kslld fields are externalized via x$kslld andv$latchname

bull Located in the PGA

RDTEX | Latch internals | RUOUG Seminar

32

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382

LATCH ATTRIBUTES

bull Each latch have at least the following attributes in kslldt

ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared

ndash PAR Is the latch Solitary or Parent for the family of child latches

ndash G2C Can two child latches be simultaneously requested in wait mode

using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92

ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above

ndash Level 0-14 Used to prevent latch deadlocks

ndash Class 0-7 Spin and wait class assigned to the latch 92 and above

RDTEX | Latch internals | RUOUG Seminar

33

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482

LATCH ATTRIBUTES BY ORACLE VERSION

RDTEX | Latch internals | RUOUG Seminar

11203 553 154 72 - 6 93

Oracle Number of latches PAR G2C LNG UFS SHARED

7340 53 14 2 3 - -

8063 80 21 7 3 - 38174 152 48 19 4 - 9

9208 242 79 37 - - 19

10202 385 114 55 - 4 47

10203 388 117 58 - 4 48

10204 394 117 59 - 4 50

11106 496 145 67 - 6 81

11107 502 145 67 - 6 83

11201 535 149 70 - 6 86

11202 551 153 72 - 6 91

34

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582

LATCH LEVELS

bull Every latch has a level in the range 0-14 to avoid deadlocks

bull Process can request a latch X in wait mode after obtaining latch Y if and

only if level X gt level Y

bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y

bull If these rules are broken the Oracle process raises

ORA-600[504] - trying to obtain another latch at incompatible level Note 281041

ORA-600[525] - trying to obtain another latch at the same level Note 1388881

ORA-600[526] - latch child order violation Note 1388891

RDTEX | Latch internals | RUOUG Seminar

35

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682

PARENT AND CHILD LATCHES

bull Latch can be a parent child or solitary latch

bull Both parent and child latches share the same latch name

bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss

bull Child latches are allocated during instance startup

bull The parent latch can be gotten independently

bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s

RDTEX | Latch internals | RUOUG Seminar

(latch info) wait_event=0 bits=10

holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl

Context saved from call 0

state=busy

36

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782

LATCH TREES

ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches

Direct SGA access program output for 9206 instance with too smallshared pool

RDTEX | Latch internals | RUOUG Seminar

ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1

waiter ospid 129 sid 72 pid 45

holding a154b7120 library cache (157) level=5 child=17 kglupc child

waiter ospid 18255 sid 65 pid 930

waiter ospid 6690 sid 554 pid 1654

waiter ospid 4685 sid 879 pid 1034

hellip

waiter ospid 29749 sid 180 pid 155

holding a154b7db8 library cache (157) level=5 child=4 kglupc child

waiter ospid 13104 sid 281 pid 220

waiter ospid 24089 sid 565 pid 636

waiter ospid 25002 sid 621 pid 1481

waiter ospid 16930 sid 1046 pid 783hellip

37

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882

LATCH CLASSES

bull Classes with different spin and wait policies

bull Up to 8 classes By default all the latches belong to class 0

bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2

bull Latch can be assigned to class by its number For example

ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000

ndash _LATCH_CLASSES=46

bull Will be discussed later

RDTEX | Latch internals | RUOUG Seminar

SQLgt select from x$ksllclass

INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4

---- ------ ----- -------- ------ ------ ------ ------ ------

0 20000 0 1 8000 8000 8000 8000 8000

hellip

6 100 12 1000 2000 3000 4000 4000 4000

38

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982

LATCH ACQUISITION IN NO-WAIT MODE

bull kslgetl(laddr0whywhere) ndash Exclusive latch get

bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep

ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches

ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches

bull IMMEDIATE_ statistics are updated

bull Used when many candidate latches are available to bypass the

latch level rule or as part of CBC chain examination

RDTEX | Latch internals | RUOUG Seminar

sqlplus nolog demo_03sql39

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082

LATCH WAITS

RDTEX | Latch internals | RUOUG Seminar 40

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182

WAITING FOR THE LATCH

RDTEX | Latch internals | RUOUG Seminar

983123 G A

C983120983125 1 C983120983125 2

983120983154983151983139983141983155983155 A

983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)

983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144

983116983137983156983139983144

983120983154983151983139983141983155983155 B

41

RDTEX | L h i l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282

LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION

bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries

again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of

times

bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep

bull A sleep corresponds to a wait for the latch free wait event

bull Initially a process sleeps for 001 s This time is doubled in every

subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip

RDTEX | Latch internals | RUOUG Seminar

Still did not found any versionwhich used this

May be it was lt= Oracle 6

42

RDTEX | L t h i t l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382

LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81

Latch gets with wait (kslgetl(laddress1hellip))

ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times

ndash Sleep on latch free event with exponential backoff

ndash Repeat

RDTEX | Latch internals | RUOUG Seminar

43

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482

8I LATCH SPIN CODE FLOW USING DTRACE

RDTEX | Latch internals | RUOUG Seminar

bull hellip Event 10046 trace

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2

983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29

983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270

sqlplus nolog demo_04_8sql 29

44

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582

POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF

bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec

RDTEX | Latch internals | RUOUG Seminar

12 ]2 )1[(

minus= +wait N

timeout

45

bull Most waits were for nothing ndash latch already was free

bull Latch utilization can not be high

bull Unnecessary spins provoke CPU thrashing

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682

92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE

sqlplus nolog demo_05sql 45

RDTEX | Latch internals | RUOUG Seminar

kslgetl(0x50006318 1)

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt kslges(0x50006318 ) -wait latch get

-gt skgslsgts(0x50006318 ) -spin latch get

-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT

-gt kskthbwt(0x0)

-gt kslwlmod() - set up Wait List

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt skgpwwait -sleep latch getsemop(11 17-10 1)

983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140

46

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782

RELIABLE LATCH WAITS

bull Hidden latch wait revolution that we missed Latch relies on waitposting

bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout

bull Exclusive latch spins 20000 cycles by default

bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock

bull Latches assigned to non-default class wait until timeout

bull Version 90 was transient and used mix of backoff and wait posting

This is why _LATCH_WAIT_POSTING parameter disappeared in9i

bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin

RDTEX | Latch internals | RUOUG Seminar

47

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882

IMAGINE THE TIME MICROSCOPE

| a a s | a

Reality One million zoomed time

1 us 1 sec

1 ms 17 min1 sec 115 days

Light speed (300000 kmsec) Sonic speed (300 msec)

CPU tick (2 GHz) 00005 sec

Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec

Max spin time for shared latch (~220005 ticks) ~10 sec

Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec

Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec

OS context switch time (10us-1ms) 10sec-17min

48

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982

LATCH WAITS UNDER TIME MICROSCOPE

Old 8i latch algorithmSpin for latch during 5 seconds

Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min

hellipSpin again for 5 seconds

Sleep again for 23 days (2 sec) hellip

New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)

If spin not succeeded go to sleep until post

According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)

| |

49

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082

_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES

bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and

_LATCH_CLASS_0 parameters

bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000

bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter

bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch

50

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182

TIMED LATCH WAIT POSTING

bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel

bull This can lead to instance hang ndash process will never be woken up

bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout

bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem

bull Generic AIX ports always use timed latch wait posting

bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux

51

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282

FINE GRAIN LATCH TUNING

bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting

bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7

bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo

microseconds using pollsys() (not semtimedop()) system call

ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times

Loop up to ldquoSpinsrdquo cycles

Yield CPU using yield() (or sched_yield())

ndash Sleep for ldquoSleepXrdquo usecs

ndash Then spin again hellip

52

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382

TRACES OF NONDEFAULT CLASS LATCH

983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152

991270

9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000

991270 E983158983141983150983156 10046 983156983154983137983139983141

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482

PERFORMANCE OF LATCH CLASSES

Row cache objects latch contention testcase on ODA (24 SMT)54

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582

IS THE _SPIN_COUNT REALLY STATIC

bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch

waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used

ldquocache buffers chainrdquo latch contention

bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass

(_LATCH_CLASS_X) By default ndash 20000

bull Shared latch spin can be tuned by _SPIN_COUNT value By

default ndash 2000 However this is not trivial

55

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682

SHARED LATCH BEHAVE LIKE ENQUEUE

bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80

bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests

bull This blocking achieved by special 0x40000000 bit in the latch

value This bit is a flag that some incompatible latch acquisitionis in progress

bull Queued processes will be woken up one by one on latch release

bull X mode latch gets effectively serialize the shared latch

bull See my blog for description of experiments

56

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782

TRACE OF SHARED LATCH WAIT

Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the

latch wait list and then spins again

983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141

983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156

983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983147983155983148983159983148983149983151983140()

983147983155983148983155983147983143983155(098316050009BAC )

983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140

983155983141983149983151983152(27)

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882

SHARED LATCH ACQUISITION

bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned

bull X mode shared latch get spins by default up to 4000 cycles

bull S mode does not spin at all (or spins in unknown way)

S mode get X mode get

Held in S mode Compatible 2_SPIN_COUNT

Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT

58

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982

LATCH RELEASE

bull Free the latch ndash kslfre(laddr)

bull Oracle process releases the latch nonatomically

bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process

bull This requires less bus invalidation and ensures propagation of latch

release to other local cachesbull Not fair policy Spinners on the local CPU board have the

preference

bull Then process posts the first process in the list of waiters

59

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082

LATCH CONTENTION

RDTEX | Latch internals | RUOUG Seminar60

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182

LATCH CONTENTION

bull Occurs when a latch is required by several processes at thesame time

bull Mostly due to bugs or abnormal application behavior

bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp

bull To diagnose the latch contention we need to investigate thelatch statistics [8]

61

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282

DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS

Latch requests arrival rate

Immediate gets efficiency

Latch sleeps ratio

Latch wait time per second

Latch spin efficiencymisses

getsspin∆

∆= _σ

misses

sleepsK

∆=

time

gets

∆=λ

gets

misses

∆= ρ

time

timewait W

∆=

_

Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics

62

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382

SAMPLED AVERAGES

By sampling of x$ksupr we can estimate

ndash Length of latch wait queue

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS

ndash Number of spinning processes in 10g

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS

Sampling of v$lathholder estimates latch utilization

Using DTrace we can measure

ndash Latch acquisition and holding times

ndash Actual spin count

w L

s N

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482

DERIVED LATCH STATISTICS

Latch utilization (PASTA)

Average holding time

Length of latch wait list

Recurrent sleeps ratio

Latch acquisition time

time

timeholdinglatchU

∆=asymp

__ ρ

Re_100

___

questsGet

TimeSnap MissGet Pct S

lowast==

λ

W L =

κ

κ σ 1minus+

)(1

W N T saq += minusλ

64

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 30: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3082

CUMULATIVE V$LATCH STATISTIC COUNTERS

RDTEX | Latch internals | RUOUG Seminar

Statistics x$ksllt When and how it increments

GETS kslltwgt ldquo++rdquo after wait mode latch get

MISSES kslltwff ldquo++rdquo after wait get if it was missed

SLEEPS kslltwsl ldquo+number_of_sleepsrdquo during get

SPIN_GETS ksllthst0 ldquo++rdquo if get was missed but not slept

WAIT_TIME kslltwtt ldquo+wait_timerdquo after latch get

IMMEDIATE_GETS kslltngt ldquo++rdquo after nowait mode latch get Is notincremented on miss

IMMEDIATE_MISSES kslltnfa ldquo++rdquo if nowait mode get was missed

30

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3182

MANY STATISTICS WERE LOST IN 102

In 102 Oracle made obsolete many useful latch statistics

Latch size was decreased and the following fields always contain 0now in v$latch

ndash v$latchwaiters_woken

ndash v$latchwaits_holding_latch

ndash v$latchsleep1hellipsleep10

ndash and many undocumented statistics

This was mentioned in 102 Reference

RDTEX | Latch internals | RUOUG Seminar

31

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3282

X$KSLLD - LATCH DESCRIPTORS

bull Flags and attributes describing each latch are stored in kslldt arrayof kslld structures

bull [K]ernel [S]ervice [L]atch [L]ock [D]escriptor

bull Some but not all of kslld fields are externalized via x$kslld andv$latchname

bull Located in the PGA

RDTEX | Latch internals | RUOUG Seminar

32

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382

LATCH ATTRIBUTES

bull Each latch have at least the following attributes in kslldt

ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared

ndash PAR Is the latch Solitary or Parent for the family of child latches

ndash G2C Can two child latches be simultaneously requested in wait mode

using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92

ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above

ndash Level 0-14 Used to prevent latch deadlocks

ndash Class 0-7 Spin and wait class assigned to the latch 92 and above

RDTEX | Latch internals | RUOUG Seminar

33

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482

LATCH ATTRIBUTES BY ORACLE VERSION

RDTEX | Latch internals | RUOUG Seminar

11203 553 154 72 - 6 93

Oracle Number of latches PAR G2C LNG UFS SHARED

7340 53 14 2 3 - -

8063 80 21 7 3 - 38174 152 48 19 4 - 9

9208 242 79 37 - - 19

10202 385 114 55 - 4 47

10203 388 117 58 - 4 48

10204 394 117 59 - 4 50

11106 496 145 67 - 6 81

11107 502 145 67 - 6 83

11201 535 149 70 - 6 86

11202 551 153 72 - 6 91

34

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582

LATCH LEVELS

bull Every latch has a level in the range 0-14 to avoid deadlocks

bull Process can request a latch X in wait mode after obtaining latch Y if and

only if level X gt level Y

bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y

bull If these rules are broken the Oracle process raises

ORA-600[504] - trying to obtain another latch at incompatible level Note 281041

ORA-600[525] - trying to obtain another latch at the same level Note 1388881

ORA-600[526] - latch child order violation Note 1388891

RDTEX | Latch internals | RUOUG Seminar

35

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682

PARENT AND CHILD LATCHES

bull Latch can be a parent child or solitary latch

bull Both parent and child latches share the same latch name

bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss

bull Child latches are allocated during instance startup

bull The parent latch can be gotten independently

bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s

RDTEX | Latch internals | RUOUG Seminar

(latch info) wait_event=0 bits=10

holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl

Context saved from call 0

state=busy

36

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782

LATCH TREES

ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches

Direct SGA access program output for 9206 instance with too smallshared pool

RDTEX | Latch internals | RUOUG Seminar

ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1

waiter ospid 129 sid 72 pid 45

holding a154b7120 library cache (157) level=5 child=17 kglupc child

waiter ospid 18255 sid 65 pid 930

waiter ospid 6690 sid 554 pid 1654

waiter ospid 4685 sid 879 pid 1034

hellip

waiter ospid 29749 sid 180 pid 155

holding a154b7db8 library cache (157) level=5 child=4 kglupc child

waiter ospid 13104 sid 281 pid 220

waiter ospid 24089 sid 565 pid 636

waiter ospid 25002 sid 621 pid 1481

waiter ospid 16930 sid 1046 pid 783hellip

37

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882

LATCH CLASSES

bull Classes with different spin and wait policies

bull Up to 8 classes By default all the latches belong to class 0

bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2

bull Latch can be assigned to class by its number For example

ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000

ndash _LATCH_CLASSES=46

bull Will be discussed later

RDTEX | Latch internals | RUOUG Seminar

SQLgt select from x$ksllclass

INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4

---- ------ ----- -------- ------ ------ ------ ------ ------

0 20000 0 1 8000 8000 8000 8000 8000

hellip

6 100 12 1000 2000 3000 4000 4000 4000

38

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982

LATCH ACQUISITION IN NO-WAIT MODE

bull kslgetl(laddr0whywhere) ndash Exclusive latch get

bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep

ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches

ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches

bull IMMEDIATE_ statistics are updated

bull Used when many candidate latches are available to bypass the

latch level rule or as part of CBC chain examination

RDTEX | Latch internals | RUOUG Seminar

sqlplus nolog demo_03sql39

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082

LATCH WAITS

RDTEX | Latch internals | RUOUG Seminar 40

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182

WAITING FOR THE LATCH

RDTEX | Latch internals | RUOUG Seminar

983123 G A

C983120983125 1 C983120983125 2

983120983154983151983139983141983155983155 A

983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)

983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144

983116983137983156983139983144

983120983154983151983139983141983155983155 B

41

RDTEX | L h i l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282

LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION

bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries

again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of

times

bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep

bull A sleep corresponds to a wait for the latch free wait event

bull Initially a process sleeps for 001 s This time is doubled in every

subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip

RDTEX | Latch internals | RUOUG Seminar

Still did not found any versionwhich used this

May be it was lt= Oracle 6

42

RDTEX | L t h i t l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382

LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81

Latch gets with wait (kslgetl(laddress1hellip))

ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times

ndash Sleep on latch free event with exponential backoff

ndash Repeat

RDTEX | Latch internals | RUOUG Seminar

43

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482

8I LATCH SPIN CODE FLOW USING DTRACE

RDTEX | Latch internals | RUOUG Seminar

bull hellip Event 10046 trace

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2

983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29

983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270

sqlplus nolog demo_04_8sql 29

44

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582

POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF

bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec

RDTEX | Latch internals | RUOUG Seminar

12 ]2 )1[(

minus= +wait N

timeout

45

bull Most waits were for nothing ndash latch already was free

bull Latch utilization can not be high

bull Unnecessary spins provoke CPU thrashing

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682

92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE

sqlplus nolog demo_05sql 45

RDTEX | Latch internals | RUOUG Seminar

kslgetl(0x50006318 1)

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt kslges(0x50006318 ) -wait latch get

-gt skgslsgts(0x50006318 ) -spin latch get

-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT

-gt kskthbwt(0x0)

-gt kslwlmod() - set up Wait List

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt skgpwwait -sleep latch getsemop(11 17-10 1)

983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140

46

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782

RELIABLE LATCH WAITS

bull Hidden latch wait revolution that we missed Latch relies on waitposting

bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout

bull Exclusive latch spins 20000 cycles by default

bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock

bull Latches assigned to non-default class wait until timeout

bull Version 90 was transient and used mix of backoff and wait posting

This is why _LATCH_WAIT_POSTING parameter disappeared in9i

bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin

RDTEX | Latch internals | RUOUG Seminar

47

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882

IMAGINE THE TIME MICROSCOPE

| a a s | a

Reality One million zoomed time

1 us 1 sec

1 ms 17 min1 sec 115 days

Light speed (300000 kmsec) Sonic speed (300 msec)

CPU tick (2 GHz) 00005 sec

Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec

Max spin time for shared latch (~220005 ticks) ~10 sec

Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec

Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec

OS context switch time (10us-1ms) 10sec-17min

48

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982

LATCH WAITS UNDER TIME MICROSCOPE

Old 8i latch algorithmSpin for latch during 5 seconds

Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min

hellipSpin again for 5 seconds

Sleep again for 23 days (2 sec) hellip

New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)

If spin not succeeded go to sleep until post

According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)

| |

49

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082

_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES

bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and

_LATCH_CLASS_0 parameters

bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000

bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter

bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch

50

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182

TIMED LATCH WAIT POSTING

bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel

bull This can lead to instance hang ndash process will never be woken up

bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout

bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem

bull Generic AIX ports always use timed latch wait posting

bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux

51

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282

FINE GRAIN LATCH TUNING

bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting

bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7

bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo

microseconds using pollsys() (not semtimedop()) system call

ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times

Loop up to ldquoSpinsrdquo cycles

Yield CPU using yield() (or sched_yield())

ndash Sleep for ldquoSleepXrdquo usecs

ndash Then spin again hellip

52

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382

TRACES OF NONDEFAULT CLASS LATCH

983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152

991270

9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000

991270 E983158983141983150983156 10046 983156983154983137983139983141

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482

PERFORMANCE OF LATCH CLASSES

Row cache objects latch contention testcase on ODA (24 SMT)54

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582

IS THE _SPIN_COUNT REALLY STATIC

bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch

waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used

ldquocache buffers chainrdquo latch contention

bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass

(_LATCH_CLASS_X) By default ndash 20000

bull Shared latch spin can be tuned by _SPIN_COUNT value By

default ndash 2000 However this is not trivial

55

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682

SHARED LATCH BEHAVE LIKE ENQUEUE

bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80

bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests

bull This blocking achieved by special 0x40000000 bit in the latch

value This bit is a flag that some incompatible latch acquisitionis in progress

bull Queued processes will be woken up one by one on latch release

bull X mode latch gets effectively serialize the shared latch

bull See my blog for description of experiments

56

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782

TRACE OF SHARED LATCH WAIT

Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the

latch wait list and then spins again

983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141

983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156

983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983147983155983148983159983148983149983151983140()

983147983155983148983155983147983143983155(098316050009BAC )

983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140

983155983141983149983151983152(27)

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882

SHARED LATCH ACQUISITION

bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned

bull X mode shared latch get spins by default up to 4000 cycles

bull S mode does not spin at all (or spins in unknown way)

S mode get X mode get

Held in S mode Compatible 2_SPIN_COUNT

Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT

58

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982

LATCH RELEASE

bull Free the latch ndash kslfre(laddr)

bull Oracle process releases the latch nonatomically

bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process

bull This requires less bus invalidation and ensures propagation of latch

release to other local cachesbull Not fair policy Spinners on the local CPU board have the

preference

bull Then process posts the first process in the list of waiters

59

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082

LATCH CONTENTION

RDTEX | Latch internals | RUOUG Seminar60

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182

LATCH CONTENTION

bull Occurs when a latch is required by several processes at thesame time

bull Mostly due to bugs or abnormal application behavior

bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp

bull To diagnose the latch contention we need to investigate thelatch statistics [8]

61

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282

DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS

Latch requests arrival rate

Immediate gets efficiency

Latch sleeps ratio

Latch wait time per second

Latch spin efficiencymisses

getsspin∆

∆= _σ

misses

sleepsK

∆=

time

gets

∆=λ

gets

misses

∆= ρ

time

timewait W

∆=

_

Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics

62

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382

SAMPLED AVERAGES

By sampling of x$ksupr we can estimate

ndash Length of latch wait queue

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS

ndash Number of spinning processes in 10g

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS

Sampling of v$lathholder estimates latch utilization

Using DTrace we can measure

ndash Latch acquisition and holding times

ndash Actual spin count

w L

s N

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482

DERIVED LATCH STATISTICS

Latch utilization (PASTA)

Average holding time

Length of latch wait list

Recurrent sleeps ratio

Latch acquisition time

time

timeholdinglatchU

∆=asymp

__ ρ

Re_100

___

questsGet

TimeSnap MissGet Pct S

lowast==

λ

W L =

κ

κ σ 1minus+

)(1

W N T saq += minusλ

64

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 31: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3182

MANY STATISTICS WERE LOST IN 102

In 102 Oracle made obsolete many useful latch statistics

Latch size was decreased and the following fields always contain 0now in v$latch

ndash v$latchwaiters_woken

ndash v$latchwaits_holding_latch

ndash v$latchsleep1hellipsleep10

ndash and many undocumented statistics

This was mentioned in 102 Reference

RDTEX | Latch internals | RUOUG Seminar

31

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3282

X$KSLLD - LATCH DESCRIPTORS

bull Flags and attributes describing each latch are stored in kslldt arrayof kslld structures

bull [K]ernel [S]ervice [L]atch [L]ock [D]escriptor

bull Some but not all of kslld fields are externalized via x$kslld andv$latchname

bull Located in the PGA

RDTEX | Latch internals | RUOUG Seminar

32

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382

LATCH ATTRIBUTES

bull Each latch have at least the following attributes in kslldt

ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared

ndash PAR Is the latch Solitary or Parent for the family of child latches

ndash G2C Can two child latches be simultaneously requested in wait mode

using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92

ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above

ndash Level 0-14 Used to prevent latch deadlocks

ndash Class 0-7 Spin and wait class assigned to the latch 92 and above

RDTEX | Latch internals | RUOUG Seminar

33

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482

LATCH ATTRIBUTES BY ORACLE VERSION

RDTEX | Latch internals | RUOUG Seminar

11203 553 154 72 - 6 93

Oracle Number of latches PAR G2C LNG UFS SHARED

7340 53 14 2 3 - -

8063 80 21 7 3 - 38174 152 48 19 4 - 9

9208 242 79 37 - - 19

10202 385 114 55 - 4 47

10203 388 117 58 - 4 48

10204 394 117 59 - 4 50

11106 496 145 67 - 6 81

11107 502 145 67 - 6 83

11201 535 149 70 - 6 86

11202 551 153 72 - 6 91

34

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582

LATCH LEVELS

bull Every latch has a level in the range 0-14 to avoid deadlocks

bull Process can request a latch X in wait mode after obtaining latch Y if and

only if level X gt level Y

bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y

bull If these rules are broken the Oracle process raises

ORA-600[504] - trying to obtain another latch at incompatible level Note 281041

ORA-600[525] - trying to obtain another latch at the same level Note 1388881

ORA-600[526] - latch child order violation Note 1388891

RDTEX | Latch internals | RUOUG Seminar

35

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682

PARENT AND CHILD LATCHES

bull Latch can be a parent child or solitary latch

bull Both parent and child latches share the same latch name

bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss

bull Child latches are allocated during instance startup

bull The parent latch can be gotten independently

bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s

RDTEX | Latch internals | RUOUG Seminar

(latch info) wait_event=0 bits=10

holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl

Context saved from call 0

state=busy

36

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782

LATCH TREES

ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches

Direct SGA access program output for 9206 instance with too smallshared pool

RDTEX | Latch internals | RUOUG Seminar

ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1

waiter ospid 129 sid 72 pid 45

holding a154b7120 library cache (157) level=5 child=17 kglupc child

waiter ospid 18255 sid 65 pid 930

waiter ospid 6690 sid 554 pid 1654

waiter ospid 4685 sid 879 pid 1034

hellip

waiter ospid 29749 sid 180 pid 155

holding a154b7db8 library cache (157) level=5 child=4 kglupc child

waiter ospid 13104 sid 281 pid 220

waiter ospid 24089 sid 565 pid 636

waiter ospid 25002 sid 621 pid 1481

waiter ospid 16930 sid 1046 pid 783hellip

37

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882

LATCH CLASSES

bull Classes with different spin and wait policies

bull Up to 8 classes By default all the latches belong to class 0

bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2

bull Latch can be assigned to class by its number For example

ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000

ndash _LATCH_CLASSES=46

bull Will be discussed later

RDTEX | Latch internals | RUOUG Seminar

SQLgt select from x$ksllclass

INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4

---- ------ ----- -------- ------ ------ ------ ------ ------

0 20000 0 1 8000 8000 8000 8000 8000

hellip

6 100 12 1000 2000 3000 4000 4000 4000

38

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982

LATCH ACQUISITION IN NO-WAIT MODE

bull kslgetl(laddr0whywhere) ndash Exclusive latch get

bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep

ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches

ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches

bull IMMEDIATE_ statistics are updated

bull Used when many candidate latches are available to bypass the

latch level rule or as part of CBC chain examination

RDTEX | Latch internals | RUOUG Seminar

sqlplus nolog demo_03sql39

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082

LATCH WAITS

RDTEX | Latch internals | RUOUG Seminar 40

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182

WAITING FOR THE LATCH

RDTEX | Latch internals | RUOUG Seminar

983123 G A

C983120983125 1 C983120983125 2

983120983154983151983139983141983155983155 A

983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)

983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144

983116983137983156983139983144

983120983154983151983139983141983155983155 B

41

RDTEX | L h i l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282

LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION

bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries

again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of

times

bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep

bull A sleep corresponds to a wait for the latch free wait event

bull Initially a process sleeps for 001 s This time is doubled in every

subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip

RDTEX | Latch internals | RUOUG Seminar

Still did not found any versionwhich used this

May be it was lt= Oracle 6

42

RDTEX | L t h i t l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382

LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81

Latch gets with wait (kslgetl(laddress1hellip))

ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times

ndash Sleep on latch free event with exponential backoff

ndash Repeat

RDTEX | Latch internals | RUOUG Seminar

43

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482

8I LATCH SPIN CODE FLOW USING DTRACE

RDTEX | Latch internals | RUOUG Seminar

bull hellip Event 10046 trace

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2

983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29

983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270

sqlplus nolog demo_04_8sql 29

44

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582

POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF

bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec

RDTEX | Latch internals | RUOUG Seminar

12 ]2 )1[(

minus= +wait N

timeout

45

bull Most waits were for nothing ndash latch already was free

bull Latch utilization can not be high

bull Unnecessary spins provoke CPU thrashing

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682

92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE

sqlplus nolog demo_05sql 45

RDTEX | Latch internals | RUOUG Seminar

kslgetl(0x50006318 1)

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt kslges(0x50006318 ) -wait latch get

-gt skgslsgts(0x50006318 ) -spin latch get

-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT

-gt kskthbwt(0x0)

-gt kslwlmod() - set up Wait List

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt skgpwwait -sleep latch getsemop(11 17-10 1)

983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140

46

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782

RELIABLE LATCH WAITS

bull Hidden latch wait revolution that we missed Latch relies on waitposting

bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout

bull Exclusive latch spins 20000 cycles by default

bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock

bull Latches assigned to non-default class wait until timeout

bull Version 90 was transient and used mix of backoff and wait posting

This is why _LATCH_WAIT_POSTING parameter disappeared in9i

bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin

RDTEX | Latch internals | RUOUG Seminar

47

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882

IMAGINE THE TIME MICROSCOPE

| a a s | a

Reality One million zoomed time

1 us 1 sec

1 ms 17 min1 sec 115 days

Light speed (300000 kmsec) Sonic speed (300 msec)

CPU tick (2 GHz) 00005 sec

Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec

Max spin time for shared latch (~220005 ticks) ~10 sec

Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec

Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec

OS context switch time (10us-1ms) 10sec-17min

48

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982

LATCH WAITS UNDER TIME MICROSCOPE

Old 8i latch algorithmSpin for latch during 5 seconds

Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min

hellipSpin again for 5 seconds

Sleep again for 23 days (2 sec) hellip

New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)

If spin not succeeded go to sleep until post

According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)

| |

49

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082

_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES

bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and

_LATCH_CLASS_0 parameters

bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000

bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter

bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch

50

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182

TIMED LATCH WAIT POSTING

bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel

bull This can lead to instance hang ndash process will never be woken up

bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout

bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem

bull Generic AIX ports always use timed latch wait posting

bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux

51

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282

FINE GRAIN LATCH TUNING

bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting

bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7

bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo

microseconds using pollsys() (not semtimedop()) system call

ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times

Loop up to ldquoSpinsrdquo cycles

Yield CPU using yield() (or sched_yield())

ndash Sleep for ldquoSleepXrdquo usecs

ndash Then spin again hellip

52

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382

TRACES OF NONDEFAULT CLASS LATCH

983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152

991270

9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000

991270 E983158983141983150983156 10046 983156983154983137983139983141

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482

PERFORMANCE OF LATCH CLASSES

Row cache objects latch contention testcase on ODA (24 SMT)54

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582

IS THE _SPIN_COUNT REALLY STATIC

bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch

waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used

ldquocache buffers chainrdquo latch contention

bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass

(_LATCH_CLASS_X) By default ndash 20000

bull Shared latch spin can be tuned by _SPIN_COUNT value By

default ndash 2000 However this is not trivial

55

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682

SHARED LATCH BEHAVE LIKE ENQUEUE

bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80

bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests

bull This blocking achieved by special 0x40000000 bit in the latch

value This bit is a flag that some incompatible latch acquisitionis in progress

bull Queued processes will be woken up one by one on latch release

bull X mode latch gets effectively serialize the shared latch

bull See my blog for description of experiments

56

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782

TRACE OF SHARED LATCH WAIT

Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the

latch wait list and then spins again

983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141

983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156

983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983147983155983148983159983148983149983151983140()

983147983155983148983155983147983143983155(098316050009BAC )

983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140

983155983141983149983151983152(27)

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882

SHARED LATCH ACQUISITION

bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned

bull X mode shared latch get spins by default up to 4000 cycles

bull S mode does not spin at all (or spins in unknown way)

S mode get X mode get

Held in S mode Compatible 2_SPIN_COUNT

Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT

58

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982

LATCH RELEASE

bull Free the latch ndash kslfre(laddr)

bull Oracle process releases the latch nonatomically

bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process

bull This requires less bus invalidation and ensures propagation of latch

release to other local cachesbull Not fair policy Spinners on the local CPU board have the

preference

bull Then process posts the first process in the list of waiters

59

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082

LATCH CONTENTION

RDTEX | Latch internals | RUOUG Seminar60

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182

LATCH CONTENTION

bull Occurs when a latch is required by several processes at thesame time

bull Mostly due to bugs or abnormal application behavior

bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp

bull To diagnose the latch contention we need to investigate thelatch statistics [8]

61

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282

DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS

Latch requests arrival rate

Immediate gets efficiency

Latch sleeps ratio

Latch wait time per second

Latch spin efficiencymisses

getsspin∆

∆= _σ

misses

sleepsK

∆=

time

gets

∆=λ

gets

misses

∆= ρ

time

timewait W

∆=

_

Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics

62

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382

SAMPLED AVERAGES

By sampling of x$ksupr we can estimate

ndash Length of latch wait queue

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS

ndash Number of spinning processes in 10g

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS

Sampling of v$lathholder estimates latch utilization

Using DTrace we can measure

ndash Latch acquisition and holding times

ndash Actual spin count

w L

s N

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482

DERIVED LATCH STATISTICS

Latch utilization (PASTA)

Average holding time

Length of latch wait list

Recurrent sleeps ratio

Latch acquisition time

time

timeholdinglatchU

∆=asymp

__ ρ

Re_100

___

questsGet

TimeSnap MissGet Pct S

lowast==

λ

W L =

κ

κ σ 1minus+

)(1

W N T saq += minusλ

64

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 32: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3282

X$KSLLD - LATCH DESCRIPTORS

bull Flags and attributes describing each latch are stored in kslldt arrayof kslld structures

bull [K]ernel [S]ervice [L]atch [L]ock [D]escriptor

bull Some but not all of kslld fields are externalized via x$kslld andv$latchname

bull Located in the PGA

RDTEX | Latch internals | RUOUG Seminar

32

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382

LATCH ATTRIBUTES

bull Each latch have at least the following attributes in kslldt

ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared

ndash PAR Is the latch Solitary or Parent for the family of child latches

ndash G2C Can two child latches be simultaneously requested in wait mode

using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92

ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above

ndash Level 0-14 Used to prevent latch deadlocks

ndash Class 0-7 Spin and wait class assigned to the latch 92 and above

RDTEX | Latch internals | RUOUG Seminar

33

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482

LATCH ATTRIBUTES BY ORACLE VERSION

RDTEX | Latch internals | RUOUG Seminar

11203 553 154 72 - 6 93

Oracle Number of latches PAR G2C LNG UFS SHARED

7340 53 14 2 3 - -

8063 80 21 7 3 - 38174 152 48 19 4 - 9

9208 242 79 37 - - 19

10202 385 114 55 - 4 47

10203 388 117 58 - 4 48

10204 394 117 59 - 4 50

11106 496 145 67 - 6 81

11107 502 145 67 - 6 83

11201 535 149 70 - 6 86

11202 551 153 72 - 6 91

34

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582

LATCH LEVELS

bull Every latch has a level in the range 0-14 to avoid deadlocks

bull Process can request a latch X in wait mode after obtaining latch Y if and

only if level X gt level Y

bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y

bull If these rules are broken the Oracle process raises

ORA-600[504] - trying to obtain another latch at incompatible level Note 281041

ORA-600[525] - trying to obtain another latch at the same level Note 1388881

ORA-600[526] - latch child order violation Note 1388891

RDTEX | Latch internals | RUOUG Seminar

35

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682

PARENT AND CHILD LATCHES

bull Latch can be a parent child or solitary latch

bull Both parent and child latches share the same latch name

bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss

bull Child latches are allocated during instance startup

bull The parent latch can be gotten independently

bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s

RDTEX | Latch internals | RUOUG Seminar

(latch info) wait_event=0 bits=10

holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl

Context saved from call 0

state=busy

36

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782

LATCH TREES

ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches

Direct SGA access program output for 9206 instance with too smallshared pool

RDTEX | Latch internals | RUOUG Seminar

ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1

waiter ospid 129 sid 72 pid 45

holding a154b7120 library cache (157) level=5 child=17 kglupc child

waiter ospid 18255 sid 65 pid 930

waiter ospid 6690 sid 554 pid 1654

waiter ospid 4685 sid 879 pid 1034

hellip

waiter ospid 29749 sid 180 pid 155

holding a154b7db8 library cache (157) level=5 child=4 kglupc child

waiter ospid 13104 sid 281 pid 220

waiter ospid 24089 sid 565 pid 636

waiter ospid 25002 sid 621 pid 1481

waiter ospid 16930 sid 1046 pid 783hellip

37

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882

LATCH CLASSES

bull Classes with different spin and wait policies

bull Up to 8 classes By default all the latches belong to class 0

bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2

bull Latch can be assigned to class by its number For example

ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000

ndash _LATCH_CLASSES=46

bull Will be discussed later

RDTEX | Latch internals | RUOUG Seminar

SQLgt select from x$ksllclass

INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4

---- ------ ----- -------- ------ ------ ------ ------ ------

0 20000 0 1 8000 8000 8000 8000 8000

hellip

6 100 12 1000 2000 3000 4000 4000 4000

38

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982

LATCH ACQUISITION IN NO-WAIT MODE

bull kslgetl(laddr0whywhere) ndash Exclusive latch get

bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep

ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches

ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches

bull IMMEDIATE_ statistics are updated

bull Used when many candidate latches are available to bypass the

latch level rule or as part of CBC chain examination

RDTEX | Latch internals | RUOUG Seminar

sqlplus nolog demo_03sql39

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082

LATCH WAITS

RDTEX | Latch internals | RUOUG Seminar 40

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182

WAITING FOR THE LATCH

RDTEX | Latch internals | RUOUG Seminar

983123 G A

C983120983125 1 C983120983125 2

983120983154983151983139983141983155983155 A

983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)

983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144

983116983137983156983139983144

983120983154983151983139983141983155983155 B

41

RDTEX | L h i l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282

LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION

bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries

again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of

times

bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep

bull A sleep corresponds to a wait for the latch free wait event

bull Initially a process sleeps for 001 s This time is doubled in every

subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip

RDTEX | Latch internals | RUOUG Seminar

Still did not found any versionwhich used this

May be it was lt= Oracle 6

42

RDTEX | L t h i t l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382

LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81

Latch gets with wait (kslgetl(laddress1hellip))

ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times

ndash Sleep on latch free event with exponential backoff

ndash Repeat

RDTEX | Latch internals | RUOUG Seminar

43

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482

8I LATCH SPIN CODE FLOW USING DTRACE

RDTEX | Latch internals | RUOUG Seminar

bull hellip Event 10046 trace

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2

983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29

983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270

sqlplus nolog demo_04_8sql 29

44

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582

POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF

bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec

RDTEX | Latch internals | RUOUG Seminar

12 ]2 )1[(

minus= +wait N

timeout

45

bull Most waits were for nothing ndash latch already was free

bull Latch utilization can not be high

bull Unnecessary spins provoke CPU thrashing

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682

92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE

sqlplus nolog demo_05sql 45

RDTEX | Latch internals | RUOUG Seminar

kslgetl(0x50006318 1)

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt kslges(0x50006318 ) -wait latch get

-gt skgslsgts(0x50006318 ) -spin latch get

-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT

-gt kskthbwt(0x0)

-gt kslwlmod() - set up Wait List

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt skgpwwait -sleep latch getsemop(11 17-10 1)

983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140

46

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782

RELIABLE LATCH WAITS

bull Hidden latch wait revolution that we missed Latch relies on waitposting

bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout

bull Exclusive latch spins 20000 cycles by default

bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock

bull Latches assigned to non-default class wait until timeout

bull Version 90 was transient and used mix of backoff and wait posting

This is why _LATCH_WAIT_POSTING parameter disappeared in9i

bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin

RDTEX | Latch internals | RUOUG Seminar

47

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882

IMAGINE THE TIME MICROSCOPE

| a a s | a

Reality One million zoomed time

1 us 1 sec

1 ms 17 min1 sec 115 days

Light speed (300000 kmsec) Sonic speed (300 msec)

CPU tick (2 GHz) 00005 sec

Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec

Max spin time for shared latch (~220005 ticks) ~10 sec

Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec

Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec

OS context switch time (10us-1ms) 10sec-17min

48

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982

LATCH WAITS UNDER TIME MICROSCOPE

Old 8i latch algorithmSpin for latch during 5 seconds

Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min

hellipSpin again for 5 seconds

Sleep again for 23 days (2 sec) hellip

New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)

If spin not succeeded go to sleep until post

According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)

| |

49

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082

_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES

bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and

_LATCH_CLASS_0 parameters

bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000

bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter

bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch

50

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182

TIMED LATCH WAIT POSTING

bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel

bull This can lead to instance hang ndash process will never be woken up

bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout

bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem

bull Generic AIX ports always use timed latch wait posting

bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux

51

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282

FINE GRAIN LATCH TUNING

bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting

bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7

bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo

microseconds using pollsys() (not semtimedop()) system call

ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times

Loop up to ldquoSpinsrdquo cycles

Yield CPU using yield() (or sched_yield())

ndash Sleep for ldquoSleepXrdquo usecs

ndash Then spin again hellip

52

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382

TRACES OF NONDEFAULT CLASS LATCH

983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152

991270

9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000

991270 E983158983141983150983156 10046 983156983154983137983139983141

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482

PERFORMANCE OF LATCH CLASSES

Row cache objects latch contention testcase on ODA (24 SMT)54

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582

IS THE _SPIN_COUNT REALLY STATIC

bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch

waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used

ldquocache buffers chainrdquo latch contention

bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass

(_LATCH_CLASS_X) By default ndash 20000

bull Shared latch spin can be tuned by _SPIN_COUNT value By

default ndash 2000 However this is not trivial

55

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682

SHARED LATCH BEHAVE LIKE ENQUEUE

bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80

bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests

bull This blocking achieved by special 0x40000000 bit in the latch

value This bit is a flag that some incompatible latch acquisitionis in progress

bull Queued processes will be woken up one by one on latch release

bull X mode latch gets effectively serialize the shared latch

bull See my blog for description of experiments

56

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782

TRACE OF SHARED LATCH WAIT

Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the

latch wait list and then spins again

983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141

983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156

983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983147983155983148983159983148983149983151983140()

983147983155983148983155983147983143983155(098316050009BAC )

983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140

983155983141983149983151983152(27)

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882

SHARED LATCH ACQUISITION

bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned

bull X mode shared latch get spins by default up to 4000 cycles

bull S mode does not spin at all (or spins in unknown way)

S mode get X mode get

Held in S mode Compatible 2_SPIN_COUNT

Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT

58

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982

LATCH RELEASE

bull Free the latch ndash kslfre(laddr)

bull Oracle process releases the latch nonatomically

bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process

bull This requires less bus invalidation and ensures propagation of latch

release to other local cachesbull Not fair policy Spinners on the local CPU board have the

preference

bull Then process posts the first process in the list of waiters

59

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082

LATCH CONTENTION

RDTEX | Latch internals | RUOUG Seminar60

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182

LATCH CONTENTION

bull Occurs when a latch is required by several processes at thesame time

bull Mostly due to bugs or abnormal application behavior

bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp

bull To diagnose the latch contention we need to investigate thelatch statistics [8]

61

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282

DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS

Latch requests arrival rate

Immediate gets efficiency

Latch sleeps ratio

Latch wait time per second

Latch spin efficiencymisses

getsspin∆

∆= _σ

misses

sleepsK

∆=

time

gets

∆=λ

gets

misses

∆= ρ

time

timewait W

∆=

_

Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics

62

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382

SAMPLED AVERAGES

By sampling of x$ksupr we can estimate

ndash Length of latch wait queue

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS

ndash Number of spinning processes in 10g

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS

Sampling of v$lathholder estimates latch utilization

Using DTrace we can measure

ndash Latch acquisition and holding times

ndash Actual spin count

w L

s N

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482

DERIVED LATCH STATISTICS

Latch utilization (PASTA)

Average holding time

Length of latch wait list

Recurrent sleeps ratio

Latch acquisition time

time

timeholdinglatchU

∆=asymp

__ ρ

Re_100

___

questsGet

TimeSnap MissGet Pct S

lowast==

λ

W L =

κ

κ σ 1minus+

)(1

W N T saq += minusλ

64

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 33: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3382

LATCH ATTRIBUTES

bull Each latch have at least the following attributes in kslldt

ndash Name Latch name as appeared in V$ views ndash SHR Is the latch Exclusive or Shared

ndash PAR Is the latch Solitary or Parent for the family of child latches

ndash G2C Can two child latches be simultaneously requested in wait mode

using kslg2c() ndash LNG Is wait posting used for this latch Obsolete since Oracle 92

ndash UFS The latch is Ultrafast It will not increment miss statistics whenSTATISTICS_LEVEL=BASIC 102 and above

ndash Level 0-14 Used to prevent latch deadlocks

ndash Class 0-7 Spin and wait class assigned to the latch 92 and above

RDTEX | Latch internals | RUOUG Seminar

33

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482

LATCH ATTRIBUTES BY ORACLE VERSION

RDTEX | Latch internals | RUOUG Seminar

11203 553 154 72 - 6 93

Oracle Number of latches PAR G2C LNG UFS SHARED

7340 53 14 2 3 - -

8063 80 21 7 3 - 38174 152 48 19 4 - 9

9208 242 79 37 - - 19

10202 385 114 55 - 4 47

10203 388 117 58 - 4 48

10204 394 117 59 - 4 50

11106 496 145 67 - 6 81

11107 502 145 67 - 6 83

11201 535 149 70 - 6 86

11202 551 153 72 - 6 91

34

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582

LATCH LEVELS

bull Every latch has a level in the range 0-14 to avoid deadlocks

bull Process can request a latch X in wait mode after obtaining latch Y if and

only if level X gt level Y

bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y

bull If these rules are broken the Oracle process raises

ORA-600[504] - trying to obtain another latch at incompatible level Note 281041

ORA-600[525] - trying to obtain another latch at the same level Note 1388881

ORA-600[526] - latch child order violation Note 1388891

RDTEX | Latch internals | RUOUG Seminar

35

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682

PARENT AND CHILD LATCHES

bull Latch can be a parent child or solitary latch

bull Both parent and child latches share the same latch name

bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss

bull Child latches are allocated during instance startup

bull The parent latch can be gotten independently

bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s

RDTEX | Latch internals | RUOUG Seminar

(latch info) wait_event=0 bits=10

holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl

Context saved from call 0

state=busy

36

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782

LATCH TREES

ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches

Direct SGA access program output for 9206 instance with too smallshared pool

RDTEX | Latch internals | RUOUG Seminar

ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1

waiter ospid 129 sid 72 pid 45

holding a154b7120 library cache (157) level=5 child=17 kglupc child

waiter ospid 18255 sid 65 pid 930

waiter ospid 6690 sid 554 pid 1654

waiter ospid 4685 sid 879 pid 1034

hellip

waiter ospid 29749 sid 180 pid 155

holding a154b7db8 library cache (157) level=5 child=4 kglupc child

waiter ospid 13104 sid 281 pid 220

waiter ospid 24089 sid 565 pid 636

waiter ospid 25002 sid 621 pid 1481

waiter ospid 16930 sid 1046 pid 783hellip

37

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882

LATCH CLASSES

bull Classes with different spin and wait policies

bull Up to 8 classes By default all the latches belong to class 0

bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2

bull Latch can be assigned to class by its number For example

ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000

ndash _LATCH_CLASSES=46

bull Will be discussed later

RDTEX | Latch internals | RUOUG Seminar

SQLgt select from x$ksllclass

INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4

---- ------ ----- -------- ------ ------ ------ ------ ------

0 20000 0 1 8000 8000 8000 8000 8000

hellip

6 100 12 1000 2000 3000 4000 4000 4000

38

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982

LATCH ACQUISITION IN NO-WAIT MODE

bull kslgetl(laddr0whywhere) ndash Exclusive latch get

bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep

ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches

ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches

bull IMMEDIATE_ statistics are updated

bull Used when many candidate latches are available to bypass the

latch level rule or as part of CBC chain examination

RDTEX | Latch internals | RUOUG Seminar

sqlplus nolog demo_03sql39

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082

LATCH WAITS

RDTEX | Latch internals | RUOUG Seminar 40

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182

WAITING FOR THE LATCH

RDTEX | Latch internals | RUOUG Seminar

983123 G A

C983120983125 1 C983120983125 2

983120983154983151983139983141983155983155 A

983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)

983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144

983116983137983156983139983144

983120983154983151983139983141983155983155 B

41

RDTEX | L h i l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282

LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION

bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries

again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of

times

bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep

bull A sleep corresponds to a wait for the latch free wait event

bull Initially a process sleeps for 001 s This time is doubled in every

subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip

RDTEX | Latch internals | RUOUG Seminar

Still did not found any versionwhich used this

May be it was lt= Oracle 6

42

RDTEX | L t h i t l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382

LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81

Latch gets with wait (kslgetl(laddress1hellip))

ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times

ndash Sleep on latch free event with exponential backoff

ndash Repeat

RDTEX | Latch internals | RUOUG Seminar

43

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482

8I LATCH SPIN CODE FLOW USING DTRACE

RDTEX | Latch internals | RUOUG Seminar

bull hellip Event 10046 trace

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2

983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29

983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270

sqlplus nolog demo_04_8sql 29

44

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582

POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF

bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec

RDTEX | Latch internals | RUOUG Seminar

12 ]2 )1[(

minus= +wait N

timeout

45

bull Most waits were for nothing ndash latch already was free

bull Latch utilization can not be high

bull Unnecessary spins provoke CPU thrashing

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682

92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE

sqlplus nolog demo_05sql 45

RDTEX | Latch internals | RUOUG Seminar

kslgetl(0x50006318 1)

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt kslges(0x50006318 ) -wait latch get

-gt skgslsgts(0x50006318 ) -spin latch get

-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT

-gt kskthbwt(0x0)

-gt kslwlmod() - set up Wait List

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt skgpwwait -sleep latch getsemop(11 17-10 1)

983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140

46

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782

RELIABLE LATCH WAITS

bull Hidden latch wait revolution that we missed Latch relies on waitposting

bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout

bull Exclusive latch spins 20000 cycles by default

bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock

bull Latches assigned to non-default class wait until timeout

bull Version 90 was transient and used mix of backoff and wait posting

This is why _LATCH_WAIT_POSTING parameter disappeared in9i

bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin

RDTEX | Latch internals | RUOUG Seminar

47

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882

IMAGINE THE TIME MICROSCOPE

| a a s | a

Reality One million zoomed time

1 us 1 sec

1 ms 17 min1 sec 115 days

Light speed (300000 kmsec) Sonic speed (300 msec)

CPU tick (2 GHz) 00005 sec

Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec

Max spin time for shared latch (~220005 ticks) ~10 sec

Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec

Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec

OS context switch time (10us-1ms) 10sec-17min

48

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982

LATCH WAITS UNDER TIME MICROSCOPE

Old 8i latch algorithmSpin for latch during 5 seconds

Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min

hellipSpin again for 5 seconds

Sleep again for 23 days (2 sec) hellip

New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)

If spin not succeeded go to sleep until post

According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)

| |

49

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082

_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES

bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and

_LATCH_CLASS_0 parameters

bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000

bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter

bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch

50

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182

TIMED LATCH WAIT POSTING

bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel

bull This can lead to instance hang ndash process will never be woken up

bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout

bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem

bull Generic AIX ports always use timed latch wait posting

bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux

51

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282

FINE GRAIN LATCH TUNING

bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting

bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7

bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo

microseconds using pollsys() (not semtimedop()) system call

ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times

Loop up to ldquoSpinsrdquo cycles

Yield CPU using yield() (or sched_yield())

ndash Sleep for ldquoSleepXrdquo usecs

ndash Then spin again hellip

52

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382

TRACES OF NONDEFAULT CLASS LATCH

983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152

991270

9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000

991270 E983158983141983150983156 10046 983156983154983137983139983141

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482

PERFORMANCE OF LATCH CLASSES

Row cache objects latch contention testcase on ODA (24 SMT)54

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582

IS THE _SPIN_COUNT REALLY STATIC

bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch

waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used

ldquocache buffers chainrdquo latch contention

bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass

(_LATCH_CLASS_X) By default ndash 20000

bull Shared latch spin can be tuned by _SPIN_COUNT value By

default ndash 2000 However this is not trivial

55

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682

SHARED LATCH BEHAVE LIKE ENQUEUE

bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80

bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests

bull This blocking achieved by special 0x40000000 bit in the latch

value This bit is a flag that some incompatible latch acquisitionis in progress

bull Queued processes will be woken up one by one on latch release

bull X mode latch gets effectively serialize the shared latch

bull See my blog for description of experiments

56

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782

TRACE OF SHARED LATCH WAIT

Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the

latch wait list and then spins again

983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141

983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156

983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983147983155983148983159983148983149983151983140()

983147983155983148983155983147983143983155(098316050009BAC )

983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140

983155983141983149983151983152(27)

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882

SHARED LATCH ACQUISITION

bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned

bull X mode shared latch get spins by default up to 4000 cycles

bull S mode does not spin at all (or spins in unknown way)

S mode get X mode get

Held in S mode Compatible 2_SPIN_COUNT

Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT

58

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982

LATCH RELEASE

bull Free the latch ndash kslfre(laddr)

bull Oracle process releases the latch nonatomically

bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process

bull This requires less bus invalidation and ensures propagation of latch

release to other local cachesbull Not fair policy Spinners on the local CPU board have the

preference

bull Then process posts the first process in the list of waiters

59

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082

LATCH CONTENTION

RDTEX | Latch internals | RUOUG Seminar60

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182

LATCH CONTENTION

bull Occurs when a latch is required by several processes at thesame time

bull Mostly due to bugs or abnormal application behavior

bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp

bull To diagnose the latch contention we need to investigate thelatch statistics [8]

61

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282

DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS

Latch requests arrival rate

Immediate gets efficiency

Latch sleeps ratio

Latch wait time per second

Latch spin efficiencymisses

getsspin∆

∆= _σ

misses

sleepsK

∆=

time

gets

∆=λ

gets

misses

∆= ρ

time

timewait W

∆=

_

Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics

62

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382

SAMPLED AVERAGES

By sampling of x$ksupr we can estimate

ndash Length of latch wait queue

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS

ndash Number of spinning processes in 10g

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS

Sampling of v$lathholder estimates latch utilization

Using DTrace we can measure

ndash Latch acquisition and holding times

ndash Actual spin count

w L

s N

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482

DERIVED LATCH STATISTICS

Latch utilization (PASTA)

Average holding time

Length of latch wait list

Recurrent sleeps ratio

Latch acquisition time

time

timeholdinglatchU

∆=asymp

__ ρ

Re_100

___

questsGet

TimeSnap MissGet Pct S

lowast==

λ

W L =

κ

κ σ 1minus+

)(1

W N T saq += minusλ

64

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 34: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3482

LATCH ATTRIBUTES BY ORACLE VERSION

RDTEX | Latch internals | RUOUG Seminar

11203 553 154 72 - 6 93

Oracle Number of latches PAR G2C LNG UFS SHARED

7340 53 14 2 3 - -

8063 80 21 7 3 - 38174 152 48 19 4 - 9

9208 242 79 37 - - 19

10202 385 114 55 - 4 47

10203 388 117 58 - 4 48

10204 394 117 59 - 4 50

11106 496 145 67 - 6 81

11107 502 145 67 - 6 83

11201 535 149 70 - 6 86

11202 551 153 72 - 6 91

34

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582

LATCH LEVELS

bull Every latch has a level in the range 0-14 to avoid deadlocks

bull Process can request a latch X in wait mode after obtaining latch Y if and

only if level X gt level Y

bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y

bull If these rules are broken the Oracle process raises

ORA-600[504] - trying to obtain another latch at incompatible level Note 281041

ORA-600[525] - trying to obtain another latch at the same level Note 1388881

ORA-600[526] - latch child order violation Note 1388891

RDTEX | Latch internals | RUOUG Seminar

35

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682

PARENT AND CHILD LATCHES

bull Latch can be a parent child or solitary latch

bull Both parent and child latches share the same latch name

bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss

bull Child latches are allocated during instance startup

bull The parent latch can be gotten independently

bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s

RDTEX | Latch internals | RUOUG Seminar

(latch info) wait_event=0 bits=10

holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl

Context saved from call 0

state=busy

36

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782

LATCH TREES

ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches

Direct SGA access program output for 9206 instance with too smallshared pool

RDTEX | Latch internals | RUOUG Seminar

ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1

waiter ospid 129 sid 72 pid 45

holding a154b7120 library cache (157) level=5 child=17 kglupc child

waiter ospid 18255 sid 65 pid 930

waiter ospid 6690 sid 554 pid 1654

waiter ospid 4685 sid 879 pid 1034

hellip

waiter ospid 29749 sid 180 pid 155

holding a154b7db8 library cache (157) level=5 child=4 kglupc child

waiter ospid 13104 sid 281 pid 220

waiter ospid 24089 sid 565 pid 636

waiter ospid 25002 sid 621 pid 1481

waiter ospid 16930 sid 1046 pid 783hellip

37

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882

LATCH CLASSES

bull Classes with different spin and wait policies

bull Up to 8 classes By default all the latches belong to class 0

bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2

bull Latch can be assigned to class by its number For example

ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000

ndash _LATCH_CLASSES=46

bull Will be discussed later

RDTEX | Latch internals | RUOUG Seminar

SQLgt select from x$ksllclass

INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4

---- ------ ----- -------- ------ ------ ------ ------ ------

0 20000 0 1 8000 8000 8000 8000 8000

hellip

6 100 12 1000 2000 3000 4000 4000 4000

38

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982

LATCH ACQUISITION IN NO-WAIT MODE

bull kslgetl(laddr0whywhere) ndash Exclusive latch get

bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep

ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches

ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches

bull IMMEDIATE_ statistics are updated

bull Used when many candidate latches are available to bypass the

latch level rule or as part of CBC chain examination

RDTEX | Latch internals | RUOUG Seminar

sqlplus nolog demo_03sql39

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082

LATCH WAITS

RDTEX | Latch internals | RUOUG Seminar 40

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182

WAITING FOR THE LATCH

RDTEX | Latch internals | RUOUG Seminar

983123 G A

C983120983125 1 C983120983125 2

983120983154983151983139983141983155983155 A

983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)

983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144

983116983137983156983139983144

983120983154983151983139983141983155983155 B

41

RDTEX | L h i l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282

LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION

bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries

again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of

times

bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep

bull A sleep corresponds to a wait for the latch free wait event

bull Initially a process sleeps for 001 s This time is doubled in every

subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip

RDTEX | Latch internals | RUOUG Seminar

Still did not found any versionwhich used this

May be it was lt= Oracle 6

42

RDTEX | L t h i t l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382

LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81

Latch gets with wait (kslgetl(laddress1hellip))

ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times

ndash Sleep on latch free event with exponential backoff

ndash Repeat

RDTEX | Latch internals | RUOUG Seminar

43

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482

8I LATCH SPIN CODE FLOW USING DTRACE

RDTEX | Latch internals | RUOUG Seminar

bull hellip Event 10046 trace

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2

983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29

983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270

sqlplus nolog demo_04_8sql 29

44

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582

POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF

bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec

RDTEX | Latch internals | RUOUG Seminar

12 ]2 )1[(

minus= +wait N

timeout

45

bull Most waits were for nothing ndash latch already was free

bull Latch utilization can not be high

bull Unnecessary spins provoke CPU thrashing

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682

92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE

sqlplus nolog demo_05sql 45

RDTEX | Latch internals | RUOUG Seminar

kslgetl(0x50006318 1)

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt kslges(0x50006318 ) -wait latch get

-gt skgslsgts(0x50006318 ) -spin latch get

-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT

-gt kskthbwt(0x0)

-gt kslwlmod() - set up Wait List

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt skgpwwait -sleep latch getsemop(11 17-10 1)

983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140

46

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782

RELIABLE LATCH WAITS

bull Hidden latch wait revolution that we missed Latch relies on waitposting

bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout

bull Exclusive latch spins 20000 cycles by default

bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock

bull Latches assigned to non-default class wait until timeout

bull Version 90 was transient and used mix of backoff and wait posting

This is why _LATCH_WAIT_POSTING parameter disappeared in9i

bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin

RDTEX | Latch internals | RUOUG Seminar

47

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882

IMAGINE THE TIME MICROSCOPE

| a a s | a

Reality One million zoomed time

1 us 1 sec

1 ms 17 min1 sec 115 days

Light speed (300000 kmsec) Sonic speed (300 msec)

CPU tick (2 GHz) 00005 sec

Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec

Max spin time for shared latch (~220005 ticks) ~10 sec

Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec

Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec

OS context switch time (10us-1ms) 10sec-17min

48

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982

LATCH WAITS UNDER TIME MICROSCOPE

Old 8i latch algorithmSpin for latch during 5 seconds

Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min

hellipSpin again for 5 seconds

Sleep again for 23 days (2 sec) hellip

New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)

If spin not succeeded go to sleep until post

According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)

| |

49

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082

_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES

bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and

_LATCH_CLASS_0 parameters

bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000

bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter

bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch

50

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182

TIMED LATCH WAIT POSTING

bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel

bull This can lead to instance hang ndash process will never be woken up

bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout

bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem

bull Generic AIX ports always use timed latch wait posting

bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux

51

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282

FINE GRAIN LATCH TUNING

bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting

bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7

bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo

microseconds using pollsys() (not semtimedop()) system call

ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times

Loop up to ldquoSpinsrdquo cycles

Yield CPU using yield() (or sched_yield())

ndash Sleep for ldquoSleepXrdquo usecs

ndash Then spin again hellip

52

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382

TRACES OF NONDEFAULT CLASS LATCH

983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152

991270

9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000

991270 E983158983141983150983156 10046 983156983154983137983139983141

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482

PERFORMANCE OF LATCH CLASSES

Row cache objects latch contention testcase on ODA (24 SMT)54

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582

IS THE _SPIN_COUNT REALLY STATIC

bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch

waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used

ldquocache buffers chainrdquo latch contention

bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass

(_LATCH_CLASS_X) By default ndash 20000

bull Shared latch spin can be tuned by _SPIN_COUNT value By

default ndash 2000 However this is not trivial

55

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682

SHARED LATCH BEHAVE LIKE ENQUEUE

bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80

bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests

bull This blocking achieved by special 0x40000000 bit in the latch

value This bit is a flag that some incompatible latch acquisitionis in progress

bull Queued processes will be woken up one by one on latch release

bull X mode latch gets effectively serialize the shared latch

bull See my blog for description of experiments

56

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782

TRACE OF SHARED LATCH WAIT

Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the

latch wait list and then spins again

983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141

983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156

983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983147983155983148983159983148983149983151983140()

983147983155983148983155983147983143983155(098316050009BAC )

983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140

983155983141983149983151983152(27)

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882

SHARED LATCH ACQUISITION

bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned

bull X mode shared latch get spins by default up to 4000 cycles

bull S mode does not spin at all (or spins in unknown way)

S mode get X mode get

Held in S mode Compatible 2_SPIN_COUNT

Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT

58

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982

LATCH RELEASE

bull Free the latch ndash kslfre(laddr)

bull Oracle process releases the latch nonatomically

bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process

bull This requires less bus invalidation and ensures propagation of latch

release to other local cachesbull Not fair policy Spinners on the local CPU board have the

preference

bull Then process posts the first process in the list of waiters

59

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082

LATCH CONTENTION

RDTEX | Latch internals | RUOUG Seminar60

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182

LATCH CONTENTION

bull Occurs when a latch is required by several processes at thesame time

bull Mostly due to bugs or abnormal application behavior

bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp

bull To diagnose the latch contention we need to investigate thelatch statistics [8]

61

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282

DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS

Latch requests arrival rate

Immediate gets efficiency

Latch sleeps ratio

Latch wait time per second

Latch spin efficiencymisses

getsspin∆

∆= _σ

misses

sleepsK

∆=

time

gets

∆=λ

gets

misses

∆= ρ

time

timewait W

∆=

_

Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics

62

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382

SAMPLED AVERAGES

By sampling of x$ksupr we can estimate

ndash Length of latch wait queue

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS

ndash Number of spinning processes in 10g

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS

Sampling of v$lathholder estimates latch utilization

Using DTrace we can measure

ndash Latch acquisition and holding times

ndash Actual spin count

w L

s N

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482

DERIVED LATCH STATISTICS

Latch utilization (PASTA)

Average holding time

Length of latch wait list

Recurrent sleeps ratio

Latch acquisition time

time

timeholdinglatchU

∆=asymp

__ ρ

Re_100

___

questsGet

TimeSnap MissGet Pct S

lowast==

λ

W L =

κ

κ σ 1minus+

)(1

W N T saq += minusλ

64

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 35: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3582

LATCH LEVELS

bull Every latch has a level in the range 0-14 to avoid deadlocks

bull Process can request a latch X in wait mode after obtaining latch Y if and

only if level X gt level Y

bull At the same level Process can request the second G2C child X in waitmode after obtaining child Y if and only if the child number of X lt childnumber of Y

bull If these rules are broken the Oracle process raises

ORA-600[504] - trying to obtain another latch at incompatible level Note 281041

ORA-600[525] - trying to obtain another latch at the same level Note 1388881

ORA-600[526] - latch child order violation Note 1388891

RDTEX | Latch internals | RUOUG Seminar

35

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682

PARENT AND CHILD LATCHES

bull Latch can be a parent child or solitary latch

bull Both parent and child latches share the same latch name

bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss

bull Child latches are allocated during instance startup

bull The parent latch can be gotten independently

bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s

RDTEX | Latch internals | RUOUG Seminar

(latch info) wait_event=0 bits=10

holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl

Context saved from call 0

state=busy

36

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782

LATCH TREES

ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches

Direct SGA access program output for 9206 instance with too smallshared pool

RDTEX | Latch internals | RUOUG Seminar

ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1

waiter ospid 129 sid 72 pid 45

holding a154b7120 library cache (157) level=5 child=17 kglupc child

waiter ospid 18255 sid 65 pid 930

waiter ospid 6690 sid 554 pid 1654

waiter ospid 4685 sid 879 pid 1034

hellip

waiter ospid 29749 sid 180 pid 155

holding a154b7db8 library cache (157) level=5 child=4 kglupc child

waiter ospid 13104 sid 281 pid 220

waiter ospid 24089 sid 565 pid 636

waiter ospid 25002 sid 621 pid 1481

waiter ospid 16930 sid 1046 pid 783hellip

37

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882

LATCH CLASSES

bull Classes with different spin and wait policies

bull Up to 8 classes By default all the latches belong to class 0

bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2

bull Latch can be assigned to class by its number For example

ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000

ndash _LATCH_CLASSES=46

bull Will be discussed later

RDTEX | Latch internals | RUOUG Seminar

SQLgt select from x$ksllclass

INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4

---- ------ ----- -------- ------ ------ ------ ------ ------

0 20000 0 1 8000 8000 8000 8000 8000

hellip

6 100 12 1000 2000 3000 4000 4000 4000

38

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982

LATCH ACQUISITION IN NO-WAIT MODE

bull kslgetl(laddr0whywhere) ndash Exclusive latch get

bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep

ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches

ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches

bull IMMEDIATE_ statistics are updated

bull Used when many candidate latches are available to bypass the

latch level rule or as part of CBC chain examination

RDTEX | Latch internals | RUOUG Seminar

sqlplus nolog demo_03sql39

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082

LATCH WAITS

RDTEX | Latch internals | RUOUG Seminar 40

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182

WAITING FOR THE LATCH

RDTEX | Latch internals | RUOUG Seminar

983123 G A

C983120983125 1 C983120983125 2

983120983154983151983139983141983155983155 A

983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)

983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144

983116983137983156983139983144

983120983154983151983139983141983155983155 B

41

RDTEX | L h i l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282

LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION

bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries

again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of

times

bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep

bull A sleep corresponds to a wait for the latch free wait event

bull Initially a process sleeps for 001 s This time is doubled in every

subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip

RDTEX | Latch internals | RUOUG Seminar

Still did not found any versionwhich used this

May be it was lt= Oracle 6

42

RDTEX | L t h i t l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382

LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81

Latch gets with wait (kslgetl(laddress1hellip))

ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times

ndash Sleep on latch free event with exponential backoff

ndash Repeat

RDTEX | Latch internals | RUOUG Seminar

43

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482

8I LATCH SPIN CODE FLOW USING DTRACE

RDTEX | Latch internals | RUOUG Seminar

bull hellip Event 10046 trace

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2

983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29

983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270

sqlplus nolog demo_04_8sql 29

44

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582

POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF

bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec

RDTEX | Latch internals | RUOUG Seminar

12 ]2 )1[(

minus= +wait N

timeout

45

bull Most waits were for nothing ndash latch already was free

bull Latch utilization can not be high

bull Unnecessary spins provoke CPU thrashing

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682

92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE

sqlplus nolog demo_05sql 45

RDTEX | Latch internals | RUOUG Seminar

kslgetl(0x50006318 1)

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt kslges(0x50006318 ) -wait latch get

-gt skgslsgts(0x50006318 ) -spin latch get

-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT

-gt kskthbwt(0x0)

-gt kslwlmod() - set up Wait List

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt skgpwwait -sleep latch getsemop(11 17-10 1)

983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140

46

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782

RELIABLE LATCH WAITS

bull Hidden latch wait revolution that we missed Latch relies on waitposting

bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout

bull Exclusive latch spins 20000 cycles by default

bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock

bull Latches assigned to non-default class wait until timeout

bull Version 90 was transient and used mix of backoff and wait posting

This is why _LATCH_WAIT_POSTING parameter disappeared in9i

bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin

RDTEX | Latch internals | RUOUG Seminar

47

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882

IMAGINE THE TIME MICROSCOPE

| a a s | a

Reality One million zoomed time

1 us 1 sec

1 ms 17 min1 sec 115 days

Light speed (300000 kmsec) Sonic speed (300 msec)

CPU tick (2 GHz) 00005 sec

Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec

Max spin time for shared latch (~220005 ticks) ~10 sec

Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec

Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec

OS context switch time (10us-1ms) 10sec-17min

48

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982

LATCH WAITS UNDER TIME MICROSCOPE

Old 8i latch algorithmSpin for latch during 5 seconds

Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min

hellipSpin again for 5 seconds

Sleep again for 23 days (2 sec) hellip

New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)

If spin not succeeded go to sleep until post

According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)

| |

49

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082

_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES

bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and

_LATCH_CLASS_0 parameters

bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000

bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter

bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch

50

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182

TIMED LATCH WAIT POSTING

bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel

bull This can lead to instance hang ndash process will never be woken up

bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout

bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem

bull Generic AIX ports always use timed latch wait posting

bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux

51

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282

FINE GRAIN LATCH TUNING

bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting

bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7

bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo

microseconds using pollsys() (not semtimedop()) system call

ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times

Loop up to ldquoSpinsrdquo cycles

Yield CPU using yield() (or sched_yield())

ndash Sleep for ldquoSleepXrdquo usecs

ndash Then spin again hellip

52

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382

TRACES OF NONDEFAULT CLASS LATCH

983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152

991270

9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000

991270 E983158983141983150983156 10046 983156983154983137983139983141

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482

PERFORMANCE OF LATCH CLASSES

Row cache objects latch contention testcase on ODA (24 SMT)54

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582

IS THE _SPIN_COUNT REALLY STATIC

bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch

waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used

ldquocache buffers chainrdquo latch contention

bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass

(_LATCH_CLASS_X) By default ndash 20000

bull Shared latch spin can be tuned by _SPIN_COUNT value By

default ndash 2000 However this is not trivial

55

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682

SHARED LATCH BEHAVE LIKE ENQUEUE

bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80

bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests

bull This blocking achieved by special 0x40000000 bit in the latch

value This bit is a flag that some incompatible latch acquisitionis in progress

bull Queued processes will be woken up one by one on latch release

bull X mode latch gets effectively serialize the shared latch

bull See my blog for description of experiments

56

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782

TRACE OF SHARED LATCH WAIT

Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the

latch wait list and then spins again

983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141

983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156

983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983147983155983148983159983148983149983151983140()

983147983155983148983155983147983143983155(098316050009BAC )

983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140

983155983141983149983151983152(27)

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882

SHARED LATCH ACQUISITION

bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned

bull X mode shared latch get spins by default up to 4000 cycles

bull S mode does not spin at all (or spins in unknown way)

S mode get X mode get

Held in S mode Compatible 2_SPIN_COUNT

Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT

58

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982

LATCH RELEASE

bull Free the latch ndash kslfre(laddr)

bull Oracle process releases the latch nonatomically

bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process

bull This requires less bus invalidation and ensures propagation of latch

release to other local cachesbull Not fair policy Spinners on the local CPU board have the

preference

bull Then process posts the first process in the list of waiters

59

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082

LATCH CONTENTION

RDTEX | Latch internals | RUOUG Seminar60

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182

LATCH CONTENTION

bull Occurs when a latch is required by several processes at thesame time

bull Mostly due to bugs or abnormal application behavior

bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp

bull To diagnose the latch contention we need to investigate thelatch statistics [8]

61

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282

DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS

Latch requests arrival rate

Immediate gets efficiency

Latch sleeps ratio

Latch wait time per second

Latch spin efficiencymisses

getsspin∆

∆= _σ

misses

sleepsK

∆=

time

gets

∆=λ

gets

misses

∆= ρ

time

timewait W

∆=

_

Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics

62

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382

SAMPLED AVERAGES

By sampling of x$ksupr we can estimate

ndash Length of latch wait queue

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS

ndash Number of spinning processes in 10g

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS

Sampling of v$lathholder estimates latch utilization

Using DTrace we can measure

ndash Latch acquisition and holding times

ndash Actual spin count

w L

s N

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482

DERIVED LATCH STATISTICS

Latch utilization (PASTA)

Average holding time

Length of latch wait list

Recurrent sleeps ratio

Latch acquisition time

time

timeholdinglatchU

∆=asymp

__ ρ

Re_100

___

questsGet

TimeSnap MissGet Pct S

lowast==

λ

W L =

κ

κ σ 1minus+

)(1

W N T saq += minusλ

64

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 36: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3682

PARENT AND CHILD LATCHES

bull Latch can be a parent child or solitary latch

bull Both parent and child latches share the same latch name

bull Parent latches are static objects in fixed SGA See x$ksmfsv and ksmss

bull Child latches are allocated during instance startup

bull The parent latch can be gotten independently

bull But may act as a master latch when acquired in special mode in kslgpl()during deadlock detection or select from v$latchx$kslltr and some x$s

RDTEX | Latch internals | RUOUG Seminar

(latch info) wait_event=0 bits=10

holding 5000a5b8 Parent+children enqueue hash chains level=4Location from where latch is held ksqcmi kslgpl

Context saved from call 0

state=busy

36

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782

LATCH TREES

ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches

Direct SGA access program output for 9206 instance with too smallshared pool

RDTEX | Latch internals | RUOUG Seminar

ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1

waiter ospid 129 sid 72 pid 45

holding a154b7120 library cache (157) level=5 child=17 kglupc child

waiter ospid 18255 sid 65 pid 930

waiter ospid 6690 sid 554 pid 1654

waiter ospid 4685 sid 879 pid 1034

hellip

waiter ospid 29749 sid 180 pid 155

holding a154b7db8 library cache (157) level=5 child=4 kglupc child

waiter ospid 13104 sid 281 pid 220

waiter ospid 24089 sid 565 pid 636

waiter ospid 25002 sid 621 pid 1481

waiter ospid 16930 sid 1046 pid 783hellip

37

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882

LATCH CLASSES

bull Classes with different spin and wait policies

bull Up to 8 classes By default all the latches belong to class 0

bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2

bull Latch can be assigned to class by its number For example

ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000

ndash _LATCH_CLASSES=46

bull Will be discussed later

RDTEX | Latch internals | RUOUG Seminar

SQLgt select from x$ksllclass

INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4

---- ------ ----- -------- ------ ------ ------ ------ ------

0 20000 0 1 8000 8000 8000 8000 8000

hellip

6 100 12 1000 2000 3000 4000 4000 4000

38

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982

LATCH ACQUISITION IN NO-WAIT MODE

bull kslgetl(laddr0whywhere) ndash Exclusive latch get

bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep

ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches

ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches

bull IMMEDIATE_ statistics are updated

bull Used when many candidate latches are available to bypass the

latch level rule or as part of CBC chain examination

RDTEX | Latch internals | RUOUG Seminar

sqlplus nolog demo_03sql39

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082

LATCH WAITS

RDTEX | Latch internals | RUOUG Seminar 40

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182

WAITING FOR THE LATCH

RDTEX | Latch internals | RUOUG Seminar

983123 G A

C983120983125 1 C983120983125 2

983120983154983151983139983141983155983155 A

983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)

983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144

983116983137983156983139983144

983120983154983151983139983141983155983155 B

41

RDTEX | L h i l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282

LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION

bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries

again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of

times

bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep

bull A sleep corresponds to a wait for the latch free wait event

bull Initially a process sleeps for 001 s This time is doubled in every

subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip

RDTEX | Latch internals | RUOUG Seminar

Still did not found any versionwhich used this

May be it was lt= Oracle 6

42

RDTEX | L t h i t l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382

LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81

Latch gets with wait (kslgetl(laddress1hellip))

ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times

ndash Sleep on latch free event with exponential backoff

ndash Repeat

RDTEX | Latch internals | RUOUG Seminar

43

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482

8I LATCH SPIN CODE FLOW USING DTRACE

RDTEX | Latch internals | RUOUG Seminar

bull hellip Event 10046 trace

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2

983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29

983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270

sqlplus nolog demo_04_8sql 29

44

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582

POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF

bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec

RDTEX | Latch internals | RUOUG Seminar

12 ]2 )1[(

minus= +wait N

timeout

45

bull Most waits were for nothing ndash latch already was free

bull Latch utilization can not be high

bull Unnecessary spins provoke CPU thrashing

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682

92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE

sqlplus nolog demo_05sql 45

RDTEX | Latch internals | RUOUG Seminar

kslgetl(0x50006318 1)

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt kslges(0x50006318 ) -wait latch get

-gt skgslsgts(0x50006318 ) -spin latch get

-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT

-gt kskthbwt(0x0)

-gt kslwlmod() - set up Wait List

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt skgpwwait -sleep latch getsemop(11 17-10 1)

983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140

46

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782

RELIABLE LATCH WAITS

bull Hidden latch wait revolution that we missed Latch relies on waitposting

bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout

bull Exclusive latch spins 20000 cycles by default

bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock

bull Latches assigned to non-default class wait until timeout

bull Version 90 was transient and used mix of backoff and wait posting

This is why _LATCH_WAIT_POSTING parameter disappeared in9i

bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin

RDTEX | Latch internals | RUOUG Seminar

47

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882

IMAGINE THE TIME MICROSCOPE

| a a s | a

Reality One million zoomed time

1 us 1 sec

1 ms 17 min1 sec 115 days

Light speed (300000 kmsec) Sonic speed (300 msec)

CPU tick (2 GHz) 00005 sec

Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec

Max spin time for shared latch (~220005 ticks) ~10 sec

Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec

Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec

OS context switch time (10us-1ms) 10sec-17min

48

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982

LATCH WAITS UNDER TIME MICROSCOPE

Old 8i latch algorithmSpin for latch during 5 seconds

Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min

hellipSpin again for 5 seconds

Sleep again for 23 days (2 sec) hellip

New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)

If spin not succeeded go to sleep until post

According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)

| |

49

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082

_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES

bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and

_LATCH_CLASS_0 parameters

bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000

bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter

bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch

50

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182

TIMED LATCH WAIT POSTING

bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel

bull This can lead to instance hang ndash process will never be woken up

bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout

bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem

bull Generic AIX ports always use timed latch wait posting

bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux

51

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282

FINE GRAIN LATCH TUNING

bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting

bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7

bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo

microseconds using pollsys() (not semtimedop()) system call

ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times

Loop up to ldquoSpinsrdquo cycles

Yield CPU using yield() (or sched_yield())

ndash Sleep for ldquoSleepXrdquo usecs

ndash Then spin again hellip

52

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382

TRACES OF NONDEFAULT CLASS LATCH

983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152

991270

9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000

991270 E983158983141983150983156 10046 983156983154983137983139983141

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482

PERFORMANCE OF LATCH CLASSES

Row cache objects latch contention testcase on ODA (24 SMT)54

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582

IS THE _SPIN_COUNT REALLY STATIC

bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch

waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used

ldquocache buffers chainrdquo latch contention

bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass

(_LATCH_CLASS_X) By default ndash 20000

bull Shared latch spin can be tuned by _SPIN_COUNT value By

default ndash 2000 However this is not trivial

55

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682

SHARED LATCH BEHAVE LIKE ENQUEUE

bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80

bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests

bull This blocking achieved by special 0x40000000 bit in the latch

value This bit is a flag that some incompatible latch acquisitionis in progress

bull Queued processes will be woken up one by one on latch release

bull X mode latch gets effectively serialize the shared latch

bull See my blog for description of experiments

56

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782

TRACE OF SHARED LATCH WAIT

Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the

latch wait list and then spins again

983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141

983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156

983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983147983155983148983159983148983149983151983140()

983147983155983148983155983147983143983155(098316050009BAC )

983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140

983155983141983149983151983152(27)

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882

SHARED LATCH ACQUISITION

bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned

bull X mode shared latch get spins by default up to 4000 cycles

bull S mode does not spin at all (or spins in unknown way)

S mode get X mode get

Held in S mode Compatible 2_SPIN_COUNT

Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT

58

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982

LATCH RELEASE

bull Free the latch ndash kslfre(laddr)

bull Oracle process releases the latch nonatomically

bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process

bull This requires less bus invalidation and ensures propagation of latch

release to other local cachesbull Not fair policy Spinners on the local CPU board have the

preference

bull Then process posts the first process in the list of waiters

59

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082

LATCH CONTENTION

RDTEX | Latch internals | RUOUG Seminar60

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182

LATCH CONTENTION

bull Occurs when a latch is required by several processes at thesame time

bull Mostly due to bugs or abnormal application behavior

bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp

bull To diagnose the latch contention we need to investigate thelatch statistics [8]

61

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282

DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS

Latch requests arrival rate

Immediate gets efficiency

Latch sleeps ratio

Latch wait time per second

Latch spin efficiencymisses

getsspin∆

∆= _σ

misses

sleepsK

∆=

time

gets

∆=λ

gets

misses

∆= ρ

time

timewait W

∆=

_

Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics

62

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382

SAMPLED AVERAGES

By sampling of x$ksupr we can estimate

ndash Length of latch wait queue

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS

ndash Number of spinning processes in 10g

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS

Sampling of v$lathholder estimates latch utilization

Using DTrace we can measure

ndash Latch acquisition and holding times

ndash Actual spin count

w L

s N

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482

DERIVED LATCH STATISTICS

Latch utilization (PASTA)

Average holding time

Length of latch wait list

Recurrent sleeps ratio

Latch acquisition time

time

timeholdinglatchU

∆=asymp

__ ρ

Re_100

___

questsGet

TimeSnap MissGet Pct S

lowast==

λ

W L =

κ

κ σ 1minus+

)(1

W N T saq += minusλ

64

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 37: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3782

LATCH TREES

ldquoRising levelrdquo rule leads to ldquotreesrdquo of processes waiting for andholding the latches

Direct SGA access program output for 9206 instance with too smallshared pool

RDTEX | Latch internals | RUOUG Seminar

ospid 28067 sid 1677 pid 61holding 3800729f0 shared pool (156) level=7 child=1 whr=1602 kghupr1

waiter ospid 129 sid 72 pid 45

holding a154b7120 library cache (157) level=5 child=17 kglupc child

waiter ospid 18255 sid 65 pid 930

waiter ospid 6690 sid 554 pid 1654

waiter ospid 4685 sid 879 pid 1034

hellip

waiter ospid 29749 sid 180 pid 155

holding a154b7db8 library cache (157) level=5 child=4 kglupc child

waiter ospid 13104 sid 281 pid 220

waiter ospid 24089 sid 565 pid 636

waiter ospid 25002 sid 621 pid 1481

waiter ospid 16930 sid 1046 pid 783hellip

37

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882

LATCH CLASSES

bull Classes with different spin and wait policies

bull Up to 8 classes By default all the latches belong to class 0

bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2

bull Latch can be assigned to class by its number For example

ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000

ndash _LATCH_CLASSES=46

bull Will be discussed later

RDTEX | Latch internals | RUOUG Seminar

SQLgt select from x$ksllclass

INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4

---- ------ ----- -------- ------ ------ ------ ------ ------

0 20000 0 1 8000 8000 8000 8000 8000

hellip

6 100 12 1000 2000 3000 4000 4000 4000

38

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982

LATCH ACQUISITION IN NO-WAIT MODE

bull kslgetl(laddr0whywhere) ndash Exclusive latch get

bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep

ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches

ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches

bull IMMEDIATE_ statistics are updated

bull Used when many candidate latches are available to bypass the

latch level rule or as part of CBC chain examination

RDTEX | Latch internals | RUOUG Seminar

sqlplus nolog demo_03sql39

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082

LATCH WAITS

RDTEX | Latch internals | RUOUG Seminar 40

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182

WAITING FOR THE LATCH

RDTEX | Latch internals | RUOUG Seminar

983123 G A

C983120983125 1 C983120983125 2

983120983154983151983139983141983155983155 A

983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)

983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144

983116983137983156983139983144

983120983154983151983139983141983155983155 B

41

RDTEX | L h i l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282

LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION

bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries

again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of

times

bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep

bull A sleep corresponds to a wait for the latch free wait event

bull Initially a process sleeps for 001 s This time is doubled in every

subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip

RDTEX | Latch internals | RUOUG Seminar

Still did not found any versionwhich used this

May be it was lt= Oracle 6

42

RDTEX | L t h i t l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382

LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81

Latch gets with wait (kslgetl(laddress1hellip))

ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times

ndash Sleep on latch free event with exponential backoff

ndash Repeat

RDTEX | Latch internals | RUOUG Seminar

43

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482

8I LATCH SPIN CODE FLOW USING DTRACE

RDTEX | Latch internals | RUOUG Seminar

bull hellip Event 10046 trace

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2

983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29

983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270

sqlplus nolog demo_04_8sql 29

44

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582

POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF

bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec

RDTEX | Latch internals | RUOUG Seminar

12 ]2 )1[(

minus= +wait N

timeout

45

bull Most waits were for nothing ndash latch already was free

bull Latch utilization can not be high

bull Unnecessary spins provoke CPU thrashing

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682

92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE

sqlplus nolog demo_05sql 45

RDTEX | Latch internals | RUOUG Seminar

kslgetl(0x50006318 1)

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt kslges(0x50006318 ) -wait latch get

-gt skgslsgts(0x50006318 ) -spin latch get

-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT

-gt kskthbwt(0x0)

-gt kslwlmod() - set up Wait List

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt skgpwwait -sleep latch getsemop(11 17-10 1)

983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140

46

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782

RELIABLE LATCH WAITS

bull Hidden latch wait revolution that we missed Latch relies on waitposting

bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout

bull Exclusive latch spins 20000 cycles by default

bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock

bull Latches assigned to non-default class wait until timeout

bull Version 90 was transient and used mix of backoff and wait posting

This is why _LATCH_WAIT_POSTING parameter disappeared in9i

bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin

RDTEX | Latch internals | RUOUG Seminar

47

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882

IMAGINE THE TIME MICROSCOPE

| a a s | a

Reality One million zoomed time

1 us 1 sec

1 ms 17 min1 sec 115 days

Light speed (300000 kmsec) Sonic speed (300 msec)

CPU tick (2 GHz) 00005 sec

Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec

Max spin time for shared latch (~220005 ticks) ~10 sec

Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec

Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec

OS context switch time (10us-1ms) 10sec-17min

48

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982

LATCH WAITS UNDER TIME MICROSCOPE

Old 8i latch algorithmSpin for latch during 5 seconds

Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min

hellipSpin again for 5 seconds

Sleep again for 23 days (2 sec) hellip

New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)

If spin not succeeded go to sleep until post

According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)

| |

49

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082

_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES

bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and

_LATCH_CLASS_0 parameters

bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000

bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter

bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch

50

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182

TIMED LATCH WAIT POSTING

bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel

bull This can lead to instance hang ndash process will never be woken up

bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout

bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem

bull Generic AIX ports always use timed latch wait posting

bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux

51

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282

FINE GRAIN LATCH TUNING

bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting

bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7

bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo

microseconds using pollsys() (not semtimedop()) system call

ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times

Loop up to ldquoSpinsrdquo cycles

Yield CPU using yield() (or sched_yield())

ndash Sleep for ldquoSleepXrdquo usecs

ndash Then spin again hellip

52

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382

TRACES OF NONDEFAULT CLASS LATCH

983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152

991270

9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000

991270 E983158983141983150983156 10046 983156983154983137983139983141

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482

PERFORMANCE OF LATCH CLASSES

Row cache objects latch contention testcase on ODA (24 SMT)54

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582

IS THE _SPIN_COUNT REALLY STATIC

bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch

waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used

ldquocache buffers chainrdquo latch contention

bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass

(_LATCH_CLASS_X) By default ndash 20000

bull Shared latch spin can be tuned by _SPIN_COUNT value By

default ndash 2000 However this is not trivial

55

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682

SHARED LATCH BEHAVE LIKE ENQUEUE

bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80

bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests

bull This blocking achieved by special 0x40000000 bit in the latch

value This bit is a flag that some incompatible latch acquisitionis in progress

bull Queued processes will be woken up one by one on latch release

bull X mode latch gets effectively serialize the shared latch

bull See my blog for description of experiments

56

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782

TRACE OF SHARED LATCH WAIT

Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the

latch wait list and then spins again

983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141

983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156

983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983147983155983148983159983148983149983151983140()

983147983155983148983155983147983143983155(098316050009BAC )

983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140

983155983141983149983151983152(27)

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882

SHARED LATCH ACQUISITION

bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned

bull X mode shared latch get spins by default up to 4000 cycles

bull S mode does not spin at all (or spins in unknown way)

S mode get X mode get

Held in S mode Compatible 2_SPIN_COUNT

Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT

58

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982

LATCH RELEASE

bull Free the latch ndash kslfre(laddr)

bull Oracle process releases the latch nonatomically

bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process

bull This requires less bus invalidation and ensures propagation of latch

release to other local cachesbull Not fair policy Spinners on the local CPU board have the

preference

bull Then process posts the first process in the list of waiters

59

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082

LATCH CONTENTION

RDTEX | Latch internals | RUOUG Seminar60

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182

LATCH CONTENTION

bull Occurs when a latch is required by several processes at thesame time

bull Mostly due to bugs or abnormal application behavior

bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp

bull To diagnose the latch contention we need to investigate thelatch statistics [8]

61

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282

DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS

Latch requests arrival rate

Immediate gets efficiency

Latch sleeps ratio

Latch wait time per second

Latch spin efficiencymisses

getsspin∆

∆= _σ

misses

sleepsK

∆=

time

gets

∆=λ

gets

misses

∆= ρ

time

timewait W

∆=

_

Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics

62

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382

SAMPLED AVERAGES

By sampling of x$ksupr we can estimate

ndash Length of latch wait queue

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS

ndash Number of spinning processes in 10g

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS

Sampling of v$lathholder estimates latch utilization

Using DTrace we can measure

ndash Latch acquisition and holding times

ndash Actual spin count

w L

s N

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482

DERIVED LATCH STATISTICS

Latch utilization (PASTA)

Average holding time

Length of latch wait list

Recurrent sleeps ratio

Latch acquisition time

time

timeholdinglatchU

∆=asymp

__ ρ

Re_100

___

questsGet

TimeSnap MissGet Pct S

lowast==

λ

W L =

κ

κ σ 1minus+

)(1

W N T saq += minusλ

64

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 38: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3882

LATCH CLASSES

bull Classes with different spin and wait policies

bull Up to 8 classes By default all the latches belong to class 0

bull Exception ndash lsquoprocess allocation latchrsquo belongs to class 2

bull Latch can be assigned to class by its number For example

ndash _LATCH_CLASS_6=100 12 1000 2000 3000 4000

ndash _LATCH_CLASSES=46

bull Will be discussed later

RDTEX | Latch internals | RUOUG Seminar

SQLgt select from x$ksllclass

INDX SPIN YIELD WAITTIME SLEEP0 SLEEP1 SLEEP2 SLEEP3 SLEEP4

---- ------ ----- -------- ------ ------ ------ ------ ------

0 20000 0 1 8000 8000 8000 8000 8000

hellip

6 100 12 1000 2000 3000 4000 4000 4000

38

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982

LATCH ACQUISITION IN NO-WAIT MODE

bull kslgetl(laddr0whywhere) ndash Exclusive latch get

bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep

ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches

ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches

bull IMMEDIATE_ statistics are updated

bull Used when many candidate latches are available to bypass the

latch level rule or as part of CBC chain examination

RDTEX | Latch internals | RUOUG Seminar

sqlplus nolog demo_03sql39

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082

LATCH WAITS

RDTEX | Latch internals | RUOUG Seminar 40

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182

WAITING FOR THE LATCH

RDTEX | Latch internals | RUOUG Seminar

983123 G A

C983120983125 1 C983120983125 2

983120983154983151983139983141983155983155 A

983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)

983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144

983116983137983156983139983144

983120983154983151983139983141983155983155 B

41

RDTEX | L h i l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282

LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION

bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries

again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of

times

bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep

bull A sleep corresponds to a wait for the latch free wait event

bull Initially a process sleeps for 001 s This time is doubled in every

subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip

RDTEX | Latch internals | RUOUG Seminar

Still did not found any versionwhich used this

May be it was lt= Oracle 6

42

RDTEX | L t h i t l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382

LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81

Latch gets with wait (kslgetl(laddress1hellip))

ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times

ndash Sleep on latch free event with exponential backoff

ndash Repeat

RDTEX | Latch internals | RUOUG Seminar

43

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482

8I LATCH SPIN CODE FLOW USING DTRACE

RDTEX | Latch internals | RUOUG Seminar

bull hellip Event 10046 trace

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2

983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29

983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270

sqlplus nolog demo_04_8sql 29

44

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582

POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF

bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec

RDTEX | Latch internals | RUOUG Seminar

12 ]2 )1[(

minus= +wait N

timeout

45

bull Most waits were for nothing ndash latch already was free

bull Latch utilization can not be high

bull Unnecessary spins provoke CPU thrashing

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682

92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE

sqlplus nolog demo_05sql 45

RDTEX | Latch internals | RUOUG Seminar

kslgetl(0x50006318 1)

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt kslges(0x50006318 ) -wait latch get

-gt skgslsgts(0x50006318 ) -spin latch get

-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT

-gt kskthbwt(0x0)

-gt kslwlmod() - set up Wait List

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt skgpwwait -sleep latch getsemop(11 17-10 1)

983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140

46

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782

RELIABLE LATCH WAITS

bull Hidden latch wait revolution that we missed Latch relies on waitposting

bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout

bull Exclusive latch spins 20000 cycles by default

bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock

bull Latches assigned to non-default class wait until timeout

bull Version 90 was transient and used mix of backoff and wait posting

This is why _LATCH_WAIT_POSTING parameter disappeared in9i

bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin

RDTEX | Latch internals | RUOUG Seminar

47

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882

IMAGINE THE TIME MICROSCOPE

| a a s | a

Reality One million zoomed time

1 us 1 sec

1 ms 17 min1 sec 115 days

Light speed (300000 kmsec) Sonic speed (300 msec)

CPU tick (2 GHz) 00005 sec

Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec

Max spin time for shared latch (~220005 ticks) ~10 sec

Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec

Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec

OS context switch time (10us-1ms) 10sec-17min

48

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982

LATCH WAITS UNDER TIME MICROSCOPE

Old 8i latch algorithmSpin for latch during 5 seconds

Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min

hellipSpin again for 5 seconds

Sleep again for 23 days (2 sec) hellip

New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)

If spin not succeeded go to sleep until post

According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)

| |

49

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082

_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES

bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and

_LATCH_CLASS_0 parameters

bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000

bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter

bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch

50

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182

TIMED LATCH WAIT POSTING

bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel

bull This can lead to instance hang ndash process will never be woken up

bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout

bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem

bull Generic AIX ports always use timed latch wait posting

bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux

51

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282

FINE GRAIN LATCH TUNING

bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting

bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7

bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo

microseconds using pollsys() (not semtimedop()) system call

ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times

Loop up to ldquoSpinsrdquo cycles

Yield CPU using yield() (or sched_yield())

ndash Sleep for ldquoSleepXrdquo usecs

ndash Then spin again hellip

52

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382

TRACES OF NONDEFAULT CLASS LATCH

983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152

991270

9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000

991270 E983158983141983150983156 10046 983156983154983137983139983141

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482

PERFORMANCE OF LATCH CLASSES

Row cache objects latch contention testcase on ODA (24 SMT)54

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582

IS THE _SPIN_COUNT REALLY STATIC

bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch

waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used

ldquocache buffers chainrdquo latch contention

bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass

(_LATCH_CLASS_X) By default ndash 20000

bull Shared latch spin can be tuned by _SPIN_COUNT value By

default ndash 2000 However this is not trivial

55

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682

SHARED LATCH BEHAVE LIKE ENQUEUE

bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80

bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests

bull This blocking achieved by special 0x40000000 bit in the latch

value This bit is a flag that some incompatible latch acquisitionis in progress

bull Queued processes will be woken up one by one on latch release

bull X mode latch gets effectively serialize the shared latch

bull See my blog for description of experiments

56

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782

TRACE OF SHARED LATCH WAIT

Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the

latch wait list and then spins again

983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141

983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156

983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983147983155983148983159983148983149983151983140()

983147983155983148983155983147983143983155(098316050009BAC )

983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140

983155983141983149983151983152(27)

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882

SHARED LATCH ACQUISITION

bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned

bull X mode shared latch get spins by default up to 4000 cycles

bull S mode does not spin at all (or spins in unknown way)

S mode get X mode get

Held in S mode Compatible 2_SPIN_COUNT

Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT

58

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982

LATCH RELEASE

bull Free the latch ndash kslfre(laddr)

bull Oracle process releases the latch nonatomically

bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process

bull This requires less bus invalidation and ensures propagation of latch

release to other local cachesbull Not fair policy Spinners on the local CPU board have the

preference

bull Then process posts the first process in the list of waiters

59

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082

LATCH CONTENTION

RDTEX | Latch internals | RUOUG Seminar60

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182

LATCH CONTENTION

bull Occurs when a latch is required by several processes at thesame time

bull Mostly due to bugs or abnormal application behavior

bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp

bull To diagnose the latch contention we need to investigate thelatch statistics [8]

61

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282

DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS

Latch requests arrival rate

Immediate gets efficiency

Latch sleeps ratio

Latch wait time per second

Latch spin efficiencymisses

getsspin∆

∆= _σ

misses

sleepsK

∆=

time

gets

∆=λ

gets

misses

∆= ρ

time

timewait W

∆=

_

Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics

62

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382

SAMPLED AVERAGES

By sampling of x$ksupr we can estimate

ndash Length of latch wait queue

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS

ndash Number of spinning processes in 10g

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS

Sampling of v$lathholder estimates latch utilization

Using DTrace we can measure

ndash Latch acquisition and holding times

ndash Actual spin count

w L

s N

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482

DERIVED LATCH STATISTICS

Latch utilization (PASTA)

Average holding time

Length of latch wait list

Recurrent sleeps ratio

Latch acquisition time

time

timeholdinglatchU

∆=asymp

__ ρ

Re_100

___

questsGet

TimeSnap MissGet Pct S

lowast==

λ

W L =

κ

κ σ 1minus+

)(1

W N T saq += minusλ

64

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 39: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 3982

LATCH ACQUISITION IN NO-WAIT MODE

bull kslgetl(laddr0whywhere) ndash Exclusive latch get

bull kslgetsl(laddr0whywheremode) ndash Shared latch getbull One attempt to get latch no spin and sleep

ndash sskgslgf() -gt uses atomic XCHGLDSTUB instructions for exclusivelatches

ndash skgslocas() -gt uses atomic CMPXCHG (CAS) for shared latches

bull IMMEDIATE_ statistics are updated

bull Used when many candidate latches are available to bypass the

latch level rule or as part of CBC chain examination

RDTEX | Latch internals | RUOUG Seminar

sqlplus nolog demo_03sql39

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082

LATCH WAITS

RDTEX | Latch internals | RUOUG Seminar 40

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182

WAITING FOR THE LATCH

RDTEX | Latch internals | RUOUG Seminar

983123 G A

C983120983125 1 C983120983125 2

983120983154983151983139983141983155983155 A

983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)

983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144

983116983137983156983139983144

983120983154983151983139983141983155983155 B

41

RDTEX | L h i l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282

LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION

bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries

again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of

times

bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep

bull A sleep corresponds to a wait for the latch free wait event

bull Initially a process sleeps for 001 s This time is doubled in every

subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip

RDTEX | Latch internals | RUOUG Seminar

Still did not found any versionwhich used this

May be it was lt= Oracle 6

42

RDTEX | L t h i t l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382

LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81

Latch gets with wait (kslgetl(laddress1hellip))

ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times

ndash Sleep on latch free event with exponential backoff

ndash Repeat

RDTEX | Latch internals | RUOUG Seminar

43

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482

8I LATCH SPIN CODE FLOW USING DTRACE

RDTEX | Latch internals | RUOUG Seminar

bull hellip Event 10046 trace

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2

983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29

983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270

sqlplus nolog demo_04_8sql 29

44

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582

POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF

bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec

RDTEX | Latch internals | RUOUG Seminar

12 ]2 )1[(

minus= +wait N

timeout

45

bull Most waits were for nothing ndash latch already was free

bull Latch utilization can not be high

bull Unnecessary spins provoke CPU thrashing

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682

92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE

sqlplus nolog demo_05sql 45

RDTEX | Latch internals | RUOUG Seminar

kslgetl(0x50006318 1)

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt kslges(0x50006318 ) -wait latch get

-gt skgslsgts(0x50006318 ) -spin latch get

-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT

-gt kskthbwt(0x0)

-gt kslwlmod() - set up Wait List

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt skgpwwait -sleep latch getsemop(11 17-10 1)

983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140

46

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782

RELIABLE LATCH WAITS

bull Hidden latch wait revolution that we missed Latch relies on waitposting

bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout

bull Exclusive latch spins 20000 cycles by default

bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock

bull Latches assigned to non-default class wait until timeout

bull Version 90 was transient and used mix of backoff and wait posting

This is why _LATCH_WAIT_POSTING parameter disappeared in9i

bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin

RDTEX | Latch internals | RUOUG Seminar

47

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882

IMAGINE THE TIME MICROSCOPE

| a a s | a

Reality One million zoomed time

1 us 1 sec

1 ms 17 min1 sec 115 days

Light speed (300000 kmsec) Sonic speed (300 msec)

CPU tick (2 GHz) 00005 sec

Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec

Max spin time for shared latch (~220005 ticks) ~10 sec

Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec

Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec

OS context switch time (10us-1ms) 10sec-17min

48

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982

LATCH WAITS UNDER TIME MICROSCOPE

Old 8i latch algorithmSpin for latch during 5 seconds

Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min

hellipSpin again for 5 seconds

Sleep again for 23 days (2 sec) hellip

New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)

If spin not succeeded go to sleep until post

According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)

| |

49

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082

_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES

bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and

_LATCH_CLASS_0 parameters

bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000

bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter

bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch

50

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182

TIMED LATCH WAIT POSTING

bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel

bull This can lead to instance hang ndash process will never be woken up

bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout

bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem

bull Generic AIX ports always use timed latch wait posting

bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux

51

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282

FINE GRAIN LATCH TUNING

bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting

bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7

bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo

microseconds using pollsys() (not semtimedop()) system call

ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times

Loop up to ldquoSpinsrdquo cycles

Yield CPU using yield() (or sched_yield())

ndash Sleep for ldquoSleepXrdquo usecs

ndash Then spin again hellip

52

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382

TRACES OF NONDEFAULT CLASS LATCH

983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152

991270

9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000

991270 E983158983141983150983156 10046 983156983154983137983139983141

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482

PERFORMANCE OF LATCH CLASSES

Row cache objects latch contention testcase on ODA (24 SMT)54

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582

IS THE _SPIN_COUNT REALLY STATIC

bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch

waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used

ldquocache buffers chainrdquo latch contention

bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass

(_LATCH_CLASS_X) By default ndash 20000

bull Shared latch spin can be tuned by _SPIN_COUNT value By

default ndash 2000 However this is not trivial

55

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682

SHARED LATCH BEHAVE LIKE ENQUEUE

bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80

bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests

bull This blocking achieved by special 0x40000000 bit in the latch

value This bit is a flag that some incompatible latch acquisitionis in progress

bull Queued processes will be woken up one by one on latch release

bull X mode latch gets effectively serialize the shared latch

bull See my blog for description of experiments

56

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782

TRACE OF SHARED LATCH WAIT

Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the

latch wait list and then spins again

983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141

983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156

983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983147983155983148983159983148983149983151983140()

983147983155983148983155983147983143983155(098316050009BAC )

983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140

983155983141983149983151983152(27)

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882

SHARED LATCH ACQUISITION

bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned

bull X mode shared latch get spins by default up to 4000 cycles

bull S mode does not spin at all (or spins in unknown way)

S mode get X mode get

Held in S mode Compatible 2_SPIN_COUNT

Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT

58

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982

LATCH RELEASE

bull Free the latch ndash kslfre(laddr)

bull Oracle process releases the latch nonatomically

bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process

bull This requires less bus invalidation and ensures propagation of latch

release to other local cachesbull Not fair policy Spinners on the local CPU board have the

preference

bull Then process posts the first process in the list of waiters

59

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082

LATCH CONTENTION

RDTEX | Latch internals | RUOUG Seminar60

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182

LATCH CONTENTION

bull Occurs when a latch is required by several processes at thesame time

bull Mostly due to bugs or abnormal application behavior

bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp

bull To diagnose the latch contention we need to investigate thelatch statistics [8]

61

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282

DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS

Latch requests arrival rate

Immediate gets efficiency

Latch sleeps ratio

Latch wait time per second

Latch spin efficiencymisses

getsspin∆

∆= _σ

misses

sleepsK

∆=

time

gets

∆=λ

gets

misses

∆= ρ

time

timewait W

∆=

_

Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics

62

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382

SAMPLED AVERAGES

By sampling of x$ksupr we can estimate

ndash Length of latch wait queue

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS

ndash Number of spinning processes in 10g

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS

Sampling of v$lathholder estimates latch utilization

Using DTrace we can measure

ndash Latch acquisition and holding times

ndash Actual spin count

w L

s N

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482

DERIVED LATCH STATISTICS

Latch utilization (PASTA)

Average holding time

Length of latch wait list

Recurrent sleeps ratio

Latch acquisition time

time

timeholdinglatchU

∆=asymp

__ ρ

Re_100

___

questsGet

TimeSnap MissGet Pct S

lowast==

λ

W L =

κ

κ σ 1minus+

)(1

W N T saq += minusλ

64

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 40: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4082

LATCH WAITS

RDTEX | Latch internals | RUOUG Seminar 40

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182

WAITING FOR THE LATCH

RDTEX | Latch internals | RUOUG Seminar

983123 G A

C983120983125 1 C983120983125 2

983120983154983151983139983141983155983155 A

983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)

983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144

983116983137983156983139983144

983120983154983151983139983141983155983155 B

41

RDTEX | L h i l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282

LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION

bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries

again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of

times

bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep

bull A sleep corresponds to a wait for the latch free wait event

bull Initially a process sleeps for 001 s This time is doubled in every

subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip

RDTEX | Latch internals | RUOUG Seminar

Still did not found any versionwhich used this

May be it was lt= Oracle 6

42

RDTEX | L t h i t l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382

LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81

Latch gets with wait (kslgetl(laddress1hellip))

ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times

ndash Sleep on latch free event with exponential backoff

ndash Repeat

RDTEX | Latch internals | RUOUG Seminar

43

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482

8I LATCH SPIN CODE FLOW USING DTRACE

RDTEX | Latch internals | RUOUG Seminar

bull hellip Event 10046 trace

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2

983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29

983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270

sqlplus nolog demo_04_8sql 29

44

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582

POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF

bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec

RDTEX | Latch internals | RUOUG Seminar

12 ]2 )1[(

minus= +wait N

timeout

45

bull Most waits were for nothing ndash latch already was free

bull Latch utilization can not be high

bull Unnecessary spins provoke CPU thrashing

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682

92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE

sqlplus nolog demo_05sql 45

RDTEX | Latch internals | RUOUG Seminar

kslgetl(0x50006318 1)

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt kslges(0x50006318 ) -wait latch get

-gt skgslsgts(0x50006318 ) -spin latch get

-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT

-gt kskthbwt(0x0)

-gt kslwlmod() - set up Wait List

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt skgpwwait -sleep latch getsemop(11 17-10 1)

983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140

46

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782

RELIABLE LATCH WAITS

bull Hidden latch wait revolution that we missed Latch relies on waitposting

bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout

bull Exclusive latch spins 20000 cycles by default

bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock

bull Latches assigned to non-default class wait until timeout

bull Version 90 was transient and used mix of backoff and wait posting

This is why _LATCH_WAIT_POSTING parameter disappeared in9i

bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin

RDTEX | Latch internals | RUOUG Seminar

47

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882

IMAGINE THE TIME MICROSCOPE

| a a s | a

Reality One million zoomed time

1 us 1 sec

1 ms 17 min1 sec 115 days

Light speed (300000 kmsec) Sonic speed (300 msec)

CPU tick (2 GHz) 00005 sec

Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec

Max spin time for shared latch (~220005 ticks) ~10 sec

Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec

Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec

OS context switch time (10us-1ms) 10sec-17min

48

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982

LATCH WAITS UNDER TIME MICROSCOPE

Old 8i latch algorithmSpin for latch during 5 seconds

Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min

hellipSpin again for 5 seconds

Sleep again for 23 days (2 sec) hellip

New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)

If spin not succeeded go to sleep until post

According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)

| |

49

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082

_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES

bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and

_LATCH_CLASS_0 parameters

bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000

bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter

bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch

50

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182

TIMED LATCH WAIT POSTING

bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel

bull This can lead to instance hang ndash process will never be woken up

bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout

bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem

bull Generic AIX ports always use timed latch wait posting

bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux

51

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282

FINE GRAIN LATCH TUNING

bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting

bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7

bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo

microseconds using pollsys() (not semtimedop()) system call

ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times

Loop up to ldquoSpinsrdquo cycles

Yield CPU using yield() (or sched_yield())

ndash Sleep for ldquoSleepXrdquo usecs

ndash Then spin again hellip

52

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382

TRACES OF NONDEFAULT CLASS LATCH

983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152

991270

9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000

991270 E983158983141983150983156 10046 983156983154983137983139983141

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482

PERFORMANCE OF LATCH CLASSES

Row cache objects latch contention testcase on ODA (24 SMT)54

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582

IS THE _SPIN_COUNT REALLY STATIC

bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch

waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used

ldquocache buffers chainrdquo latch contention

bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass

(_LATCH_CLASS_X) By default ndash 20000

bull Shared latch spin can be tuned by _SPIN_COUNT value By

default ndash 2000 However this is not trivial

55

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682

SHARED LATCH BEHAVE LIKE ENQUEUE

bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80

bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests

bull This blocking achieved by special 0x40000000 bit in the latch

value This bit is a flag that some incompatible latch acquisitionis in progress

bull Queued processes will be woken up one by one on latch release

bull X mode latch gets effectively serialize the shared latch

bull See my blog for description of experiments

56

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782

TRACE OF SHARED LATCH WAIT

Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the

latch wait list and then spins again

983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141

983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156

983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983147983155983148983159983148983149983151983140()

983147983155983148983155983147983143983155(098316050009BAC )

983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140

983155983141983149983151983152(27)

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882

SHARED LATCH ACQUISITION

bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned

bull X mode shared latch get spins by default up to 4000 cycles

bull S mode does not spin at all (or spins in unknown way)

S mode get X mode get

Held in S mode Compatible 2_SPIN_COUNT

Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT

58

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982

LATCH RELEASE

bull Free the latch ndash kslfre(laddr)

bull Oracle process releases the latch nonatomically

bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process

bull This requires less bus invalidation and ensures propagation of latch

release to other local cachesbull Not fair policy Spinners on the local CPU board have the

preference

bull Then process posts the first process in the list of waiters

59

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082

LATCH CONTENTION

RDTEX | Latch internals | RUOUG Seminar60

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182

LATCH CONTENTION

bull Occurs when a latch is required by several processes at thesame time

bull Mostly due to bugs or abnormal application behavior

bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp

bull To diagnose the latch contention we need to investigate thelatch statistics [8]

61

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282

DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS

Latch requests arrival rate

Immediate gets efficiency

Latch sleeps ratio

Latch wait time per second

Latch spin efficiencymisses

getsspin∆

∆= _σ

misses

sleepsK

∆=

time

gets

∆=λ

gets

misses

∆= ρ

time

timewait W

∆=

_

Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics

62

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382

SAMPLED AVERAGES

By sampling of x$ksupr we can estimate

ndash Length of latch wait queue

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS

ndash Number of spinning processes in 10g

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS

Sampling of v$lathholder estimates latch utilization

Using DTrace we can measure

ndash Latch acquisition and holding times

ndash Actual spin count

w L

s N

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482

DERIVED LATCH STATISTICS

Latch utilization (PASTA)

Average holding time

Length of latch wait list

Recurrent sleeps ratio

Latch acquisition time

time

timeholdinglatchU

∆=asymp

__ ρ

Re_100

___

questsGet

TimeSnap MissGet Pct S

lowast==

λ

W L =

κ

κ σ 1minus+

)(1

W N T saq += minusλ

64

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 41: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4182

WAITING FOR THE LATCH

RDTEX | Latch internals | RUOUG Seminar

983123 G A

C983120983125 1 C983120983125 2

983120983154983151983139983141983155983155 A

983120983154983151983139983141983155983155 B 983159983137983145983156983155(983155983152983145983150983155 983137983150983140 983155983148983141983141983152983155)

983120983154983151983139983141983155983155 A983144983151983148983140983155 983137 983148983137983156983139983144

983116983137983156983139983144

983120983154983151983139983141983155983155 B

41

RDTEX | L h i l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282

LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION

bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries

again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of

times

bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep

bull A sleep corresponds to a wait for the latch free wait event

bull Initially a process sleeps for 001 s This time is doubled in every

subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip

RDTEX | Latch internals | RUOUG Seminar

Still did not found any versionwhich used this

May be it was lt= Oracle 6

42

RDTEX | L t h i t l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382

LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81

Latch gets with wait (kslgetl(laddress1hellip))

ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times

ndash Sleep on latch free event with exponential backoff

ndash Repeat

RDTEX | Latch internals | RUOUG Seminar

43

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482

8I LATCH SPIN CODE FLOW USING DTRACE

RDTEX | Latch internals | RUOUG Seminar

bull hellip Event 10046 trace

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2

983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29

983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270

sqlplus nolog demo_04_8sql 29

44

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582

POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF

bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec

RDTEX | Latch internals | RUOUG Seminar

12 ]2 )1[(

minus= +wait N

timeout

45

bull Most waits were for nothing ndash latch already was free

bull Latch utilization can not be high

bull Unnecessary spins provoke CPU thrashing

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682

92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE

sqlplus nolog demo_05sql 45

RDTEX | Latch internals | RUOUG Seminar

kslgetl(0x50006318 1)

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt kslges(0x50006318 ) -wait latch get

-gt skgslsgts(0x50006318 ) -spin latch get

-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT

-gt kskthbwt(0x0)

-gt kslwlmod() - set up Wait List

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt skgpwwait -sleep latch getsemop(11 17-10 1)

983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140

46

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782

RELIABLE LATCH WAITS

bull Hidden latch wait revolution that we missed Latch relies on waitposting

bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout

bull Exclusive latch spins 20000 cycles by default

bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock

bull Latches assigned to non-default class wait until timeout

bull Version 90 was transient and used mix of backoff and wait posting

This is why _LATCH_WAIT_POSTING parameter disappeared in9i

bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin

RDTEX | Latch internals | RUOUG Seminar

47

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882

IMAGINE THE TIME MICROSCOPE

| a a s | a

Reality One million zoomed time

1 us 1 sec

1 ms 17 min1 sec 115 days

Light speed (300000 kmsec) Sonic speed (300 msec)

CPU tick (2 GHz) 00005 sec

Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec

Max spin time for shared latch (~220005 ticks) ~10 sec

Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec

Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec

OS context switch time (10us-1ms) 10sec-17min

48

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982

LATCH WAITS UNDER TIME MICROSCOPE

Old 8i latch algorithmSpin for latch during 5 seconds

Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min

hellipSpin again for 5 seconds

Sleep again for 23 days (2 sec) hellip

New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)

If spin not succeeded go to sleep until post

According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)

| |

49

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082

_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES

bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and

_LATCH_CLASS_0 parameters

bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000

bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter

bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch

50

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182

TIMED LATCH WAIT POSTING

bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel

bull This can lead to instance hang ndash process will never be woken up

bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout

bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem

bull Generic AIX ports always use timed latch wait posting

bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux

51

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282

FINE GRAIN LATCH TUNING

bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting

bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7

bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo

microseconds using pollsys() (not semtimedop()) system call

ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times

Loop up to ldquoSpinsrdquo cycles

Yield CPU using yield() (or sched_yield())

ndash Sleep for ldquoSleepXrdquo usecs

ndash Then spin again hellip

52

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382

TRACES OF NONDEFAULT CLASS LATCH

983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152

991270

9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000

991270 E983158983141983150983156 10046 983156983154983137983139983141

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482

PERFORMANCE OF LATCH CLASSES

Row cache objects latch contention testcase on ODA (24 SMT)54

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582

IS THE _SPIN_COUNT REALLY STATIC

bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch

waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used

ldquocache buffers chainrdquo latch contention

bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass

(_LATCH_CLASS_X) By default ndash 20000

bull Shared latch spin can be tuned by _SPIN_COUNT value By

default ndash 2000 However this is not trivial

55

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682

SHARED LATCH BEHAVE LIKE ENQUEUE

bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80

bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests

bull This blocking achieved by special 0x40000000 bit in the latch

value This bit is a flag that some incompatible latch acquisitionis in progress

bull Queued processes will be woken up one by one on latch release

bull X mode latch gets effectively serialize the shared latch

bull See my blog for description of experiments

56

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782

TRACE OF SHARED LATCH WAIT

Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the

latch wait list and then spins again

983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141

983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156

983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983147983155983148983159983148983149983151983140()

983147983155983148983155983147983143983155(098316050009BAC )

983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140

983155983141983149983151983152(27)

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882

SHARED LATCH ACQUISITION

bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned

bull X mode shared latch get spins by default up to 4000 cycles

bull S mode does not spin at all (or spins in unknown way)

S mode get X mode get

Held in S mode Compatible 2_SPIN_COUNT

Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT

58

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982

LATCH RELEASE

bull Free the latch ndash kslfre(laddr)

bull Oracle process releases the latch nonatomically

bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process

bull This requires less bus invalidation and ensures propagation of latch

release to other local cachesbull Not fair policy Spinners on the local CPU board have the

preference

bull Then process posts the first process in the list of waiters

59

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082

LATCH CONTENTION

RDTEX | Latch internals | RUOUG Seminar60

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182

LATCH CONTENTION

bull Occurs when a latch is required by several processes at thesame time

bull Mostly due to bugs or abnormal application behavior

bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp

bull To diagnose the latch contention we need to investigate thelatch statistics [8]

61

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282

DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS

Latch requests arrival rate

Immediate gets efficiency

Latch sleeps ratio

Latch wait time per second

Latch spin efficiencymisses

getsspin∆

∆= _σ

misses

sleepsK

∆=

time

gets

∆=λ

gets

misses

∆= ρ

time

timewait W

∆=

_

Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics

62

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382

SAMPLED AVERAGES

By sampling of x$ksupr we can estimate

ndash Length of latch wait queue

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS

ndash Number of spinning processes in 10g

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS

Sampling of v$lathholder estimates latch utilization

Using DTrace we can measure

ndash Latch acquisition and holding times

ndash Actual spin count

w L

s N

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482

DERIVED LATCH STATISTICS

Latch utilization (PASTA)

Average holding time

Length of latch wait list

Recurrent sleeps ratio

Latch acquisition time

time

timeholdinglatchU

∆=asymp

__ ρ

Re_100

___

questsGet

TimeSnap MissGet Pct S

lowast==

λ

W L =

κ

κ σ 1minus+

)(1

W N T saq += minusλ

64

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 42: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4282

LATCH ACQUISITION IN WAIT MODEOFFICIAL VERSION

bull hellip If a required latch is busy the process requesting it spins(counts up to a large number (_LATCH_SPIN_COUNT)) tries

again and if still not available spins againbull This loop is repeated up to _SPIN_COUNT maximum number of

times

bull If after that entire loop the latch is still not available the processmust yield the CPU and go to sleep

bull A sleep corresponds to a wait for the latch free wait event

bull Initially a process sleeps for 001 s This time is doubled in every

subsequent sleep up to a maximum determined by the parameter _MAX_EXPONENTIAL_SLEEP (02s or 2s)hellip

RDTEX | Latch internals | RUOUG Seminar

Still did not found any versionwhich used this

May be it was lt= Oracle 6

42

RDTEX | L t h i t l | RUOUG S i

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382

LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81

Latch gets with wait (kslgetl(laddress1hellip))

ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times

ndash Sleep on latch free event with exponential backoff

ndash Repeat

RDTEX | Latch internals | RUOUG Seminar

43

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482

8I LATCH SPIN CODE FLOW USING DTRACE

RDTEX | Latch internals | RUOUG Seminar

bull hellip Event 10046 trace

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2

983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29

983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270

sqlplus nolog demo_04_8sql 29

44

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582

POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF

bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec

RDTEX | Latch internals | RUOUG Seminar

12 ]2 )1[(

minus= +wait N

timeout

45

bull Most waits were for nothing ndash latch already was free

bull Latch utilization can not be high

bull Unnecessary spins provoke CPU thrashing

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682

92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE

sqlplus nolog demo_05sql 45

RDTEX | Latch internals | RUOUG Seminar

kslgetl(0x50006318 1)

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt kslges(0x50006318 ) -wait latch get

-gt skgslsgts(0x50006318 ) -spin latch get

-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT

-gt kskthbwt(0x0)

-gt kslwlmod() - set up Wait List

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt skgpwwait -sleep latch getsemop(11 17-10 1)

983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140

46

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782

RELIABLE LATCH WAITS

bull Hidden latch wait revolution that we missed Latch relies on waitposting

bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout

bull Exclusive latch spins 20000 cycles by default

bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock

bull Latches assigned to non-default class wait until timeout

bull Version 90 was transient and used mix of backoff and wait posting

This is why _LATCH_WAIT_POSTING parameter disappeared in9i

bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin

RDTEX | Latch internals | RUOUG Seminar

47

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882

IMAGINE THE TIME MICROSCOPE

| a a s | a

Reality One million zoomed time

1 us 1 sec

1 ms 17 min1 sec 115 days

Light speed (300000 kmsec) Sonic speed (300 msec)

CPU tick (2 GHz) 00005 sec

Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec

Max spin time for shared latch (~220005 ticks) ~10 sec

Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec

Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec

OS context switch time (10us-1ms) 10sec-17min

48

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982

LATCH WAITS UNDER TIME MICROSCOPE

Old 8i latch algorithmSpin for latch during 5 seconds

Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min

hellipSpin again for 5 seconds

Sleep again for 23 days (2 sec) hellip

New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)

If spin not succeeded go to sleep until post

According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)

| |

49

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082

_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES

bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and

_LATCH_CLASS_0 parameters

bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000

bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter

bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch

50

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182

TIMED LATCH WAIT POSTING

bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel

bull This can lead to instance hang ndash process will never be woken up

bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout

bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem

bull Generic AIX ports always use timed latch wait posting

bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux

51

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282

FINE GRAIN LATCH TUNING

bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting

bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7

bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo

microseconds using pollsys() (not semtimedop()) system call

ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times

Loop up to ldquoSpinsrdquo cycles

Yield CPU using yield() (or sched_yield())

ndash Sleep for ldquoSleepXrdquo usecs

ndash Then spin again hellip

52

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382

TRACES OF NONDEFAULT CLASS LATCH

983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152

991270

9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000

991270 E983158983141983150983156 10046 983156983154983137983139983141

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482

PERFORMANCE OF LATCH CLASSES

Row cache objects latch contention testcase on ODA (24 SMT)54

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582

IS THE _SPIN_COUNT REALLY STATIC

bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch

waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used

ldquocache buffers chainrdquo latch contention

bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass

(_LATCH_CLASS_X) By default ndash 20000

bull Shared latch spin can be tuned by _SPIN_COUNT value By

default ndash 2000 However this is not trivial

55

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682

SHARED LATCH BEHAVE LIKE ENQUEUE

bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80

bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests

bull This blocking achieved by special 0x40000000 bit in the latch

value This bit is a flag that some incompatible latch acquisitionis in progress

bull Queued processes will be woken up one by one on latch release

bull X mode latch gets effectively serialize the shared latch

bull See my blog for description of experiments

56

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782

TRACE OF SHARED LATCH WAIT

Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the

latch wait list and then spins again

983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141

983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156

983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983147983155983148983159983148983149983151983140()

983147983155983148983155983147983143983155(098316050009BAC )

983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140

983155983141983149983151983152(27)

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882

SHARED LATCH ACQUISITION

bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned

bull X mode shared latch get spins by default up to 4000 cycles

bull S mode does not spin at all (or spins in unknown way)

S mode get X mode get

Held in S mode Compatible 2_SPIN_COUNT

Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT

58

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982

LATCH RELEASE

bull Free the latch ndash kslfre(laddr)

bull Oracle process releases the latch nonatomically

bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process

bull This requires less bus invalidation and ensures propagation of latch

release to other local cachesbull Not fair policy Spinners on the local CPU board have the

preference

bull Then process posts the first process in the list of waiters

59

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082

LATCH CONTENTION

RDTEX | Latch internals | RUOUG Seminar60

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182

LATCH CONTENTION

bull Occurs when a latch is required by several processes at thesame time

bull Mostly due to bugs or abnormal application behavior

bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp

bull To diagnose the latch contention we need to investigate thelatch statistics [8]

61

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282

DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS

Latch requests arrival rate

Immediate gets efficiency

Latch sleeps ratio

Latch wait time per second

Latch spin efficiencymisses

getsspin∆

∆= _σ

misses

sleepsK

∆=

time

gets

∆=λ

gets

misses

∆= ρ

time

timewait W

∆=

_

Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics

62

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382

SAMPLED AVERAGES

By sampling of x$ksupr we can estimate

ndash Length of latch wait queue

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS

ndash Number of spinning processes in 10g

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS

Sampling of v$lathholder estimates latch utilization

Using DTrace we can measure

ndash Latch acquisition and holding times

ndash Actual spin count

w L

s N

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482

DERIVED LATCH STATISTICS

Latch utilization (PASTA)

Average holding time

Length of latch wait list

Recurrent sleeps ratio

Latch acquisition time

time

timeholdinglatchU

∆=asymp

__ ρ

Re_100

___

questsGet

TimeSnap MissGet Pct S

lowast==

λ

W L =

κ

κ σ 1minus+

)(1

W N T saq += minusλ

64

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 43: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4382

LATCH ACQUISITION IN WAIT MODEANOTHER OFFICIAL VERSION WAS USED IN 73-81

Latch gets with wait (kslgetl(laddress1hellip))

ndash One fast get no spin (sskgslgf()) ndash Spin get check the latch upto _SPIN_COUNT times

ndash Sleep on latch free event with exponential backoff

ndash Repeat

RDTEX | Latch internals | RUOUG Seminar

43

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482

8I LATCH SPIN CODE FLOW USING DTRACE

RDTEX | Latch internals | RUOUG Seminar

bull hellip Event 10046 trace

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2

983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29

983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270

sqlplus nolog demo_04_8sql 29

44

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582

POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF

bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec

RDTEX | Latch internals | RUOUG Seminar

12 ]2 )1[(

minus= +wait N

timeout

45

bull Most waits were for nothing ndash latch already was free

bull Latch utilization can not be high

bull Unnecessary spins provoke CPU thrashing

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682

92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE

sqlplus nolog demo_05sql 45

RDTEX | Latch internals | RUOUG Seminar

kslgetl(0x50006318 1)

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt kslges(0x50006318 ) -wait latch get

-gt skgslsgts(0x50006318 ) -spin latch get

-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT

-gt kskthbwt(0x0)

-gt kslwlmod() - set up Wait List

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt skgpwwait -sleep latch getsemop(11 17-10 1)

983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140

46

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782

RELIABLE LATCH WAITS

bull Hidden latch wait revolution that we missed Latch relies on waitposting

bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout

bull Exclusive latch spins 20000 cycles by default

bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock

bull Latches assigned to non-default class wait until timeout

bull Version 90 was transient and used mix of backoff and wait posting

This is why _LATCH_WAIT_POSTING parameter disappeared in9i

bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin

RDTEX | Latch internals | RUOUG Seminar

47

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882

IMAGINE THE TIME MICROSCOPE

| a a s | a

Reality One million zoomed time

1 us 1 sec

1 ms 17 min1 sec 115 days

Light speed (300000 kmsec) Sonic speed (300 msec)

CPU tick (2 GHz) 00005 sec

Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec

Max spin time for shared latch (~220005 ticks) ~10 sec

Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec

Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec

OS context switch time (10us-1ms) 10sec-17min

48

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982

LATCH WAITS UNDER TIME MICROSCOPE

Old 8i latch algorithmSpin for latch during 5 seconds

Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min

hellipSpin again for 5 seconds

Sleep again for 23 days (2 sec) hellip

New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)

If spin not succeeded go to sleep until post

According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)

| |

49

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082

_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES

bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and

_LATCH_CLASS_0 parameters

bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000

bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter

bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch

50

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182

TIMED LATCH WAIT POSTING

bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel

bull This can lead to instance hang ndash process will never be woken up

bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout

bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem

bull Generic AIX ports always use timed latch wait posting

bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux

51

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282

FINE GRAIN LATCH TUNING

bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting

bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7

bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo

microseconds using pollsys() (not semtimedop()) system call

ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times

Loop up to ldquoSpinsrdquo cycles

Yield CPU using yield() (or sched_yield())

ndash Sleep for ldquoSleepXrdquo usecs

ndash Then spin again hellip

52

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382

TRACES OF NONDEFAULT CLASS LATCH

983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152

991270

9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000

991270 E983158983141983150983156 10046 983156983154983137983139983141

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482

PERFORMANCE OF LATCH CLASSES

Row cache objects latch contention testcase on ODA (24 SMT)54

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582

IS THE _SPIN_COUNT REALLY STATIC

bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch

waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used

ldquocache buffers chainrdquo latch contention

bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass

(_LATCH_CLASS_X) By default ndash 20000

bull Shared latch spin can be tuned by _SPIN_COUNT value By

default ndash 2000 However this is not trivial

55

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682

SHARED LATCH BEHAVE LIKE ENQUEUE

bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80

bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests

bull This blocking achieved by special 0x40000000 bit in the latch

value This bit is a flag that some incompatible latch acquisitionis in progress

bull Queued processes will be woken up one by one on latch release

bull X mode latch gets effectively serialize the shared latch

bull See my blog for description of experiments

56

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782

TRACE OF SHARED LATCH WAIT

Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the

latch wait list and then spins again

983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141

983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156

983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983147983155983148983159983148983149983151983140()

983147983155983148983155983147983143983155(098316050009BAC )

983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140

983155983141983149983151983152(27)

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882

SHARED LATCH ACQUISITION

bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned

bull X mode shared latch get spins by default up to 4000 cycles

bull S mode does not spin at all (or spins in unknown way)

S mode get X mode get

Held in S mode Compatible 2_SPIN_COUNT

Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT

58

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982

LATCH RELEASE

bull Free the latch ndash kslfre(laddr)

bull Oracle process releases the latch nonatomically

bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process

bull This requires less bus invalidation and ensures propagation of latch

release to other local cachesbull Not fair policy Spinners on the local CPU board have the

preference

bull Then process posts the first process in the list of waiters

59

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082

LATCH CONTENTION

RDTEX | Latch internals | RUOUG Seminar60

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182

LATCH CONTENTION

bull Occurs when a latch is required by several processes at thesame time

bull Mostly due to bugs or abnormal application behavior

bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp

bull To diagnose the latch contention we need to investigate thelatch statistics [8]

61

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282

DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS

Latch requests arrival rate

Immediate gets efficiency

Latch sleeps ratio

Latch wait time per second

Latch spin efficiencymisses

getsspin∆

∆= _σ

misses

sleepsK

∆=

time

gets

∆=λ

gets

misses

∆= ρ

time

timewait W

∆=

_

Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics

62

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382

SAMPLED AVERAGES

By sampling of x$ksupr we can estimate

ndash Length of latch wait queue

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS

ndash Number of spinning processes in 10g

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS

Sampling of v$lathholder estimates latch utilization

Using DTrace we can measure

ndash Latch acquisition and holding times

ndash Actual spin count

w L

s N

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482

DERIVED LATCH STATISTICS

Latch utilization (PASTA)

Average holding time

Length of latch wait list

Recurrent sleeps ratio

Latch acquisition time

time

timeholdinglatchU

∆=asymp

__ ρ

Re_100

___

questsGet

TimeSnap MissGet Pct S

lowast==

λ

W L =

κ

κ σ 1minus+

)(1

W N T saq += minusλ

64

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 44: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4482

8I LATCH SPIN CODE FLOW USING DTRACE

RDTEX | Latch internals | RUOUG Seminar

bull hellip Event 10046 trace

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=0bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=1

bull WAIT 0 nam=latch free ela= 0 p1=536893688 p2=29 p3=2

983147983155983148983143983141983156983148(0983160200058F8123) 983085 K983123983116 GE983124 983141983160983139983148983157983155983145983158983141 983116983137983156983139983144 29

983147983155983148983143983141983155(0983160200058F8 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155 = 983123983120I983118983135C983119983125983118983124

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 1

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 2

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=10 983149983155) 983085 983123983148983141983141983152 3

983155983147983143983155983148983156983155983156(0983160200058F8) 983139983137983148983148 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983152983151983148983148983155983161983155(983156983145983149983141983151983157983156=30 983149983155) 983085 983123983148983141983141983152 4 991270

sqlplus nolog demo_04_8sql 29

44

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582

POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF

bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec

RDTEX | Latch internals | RUOUG Seminar

12 ]2 )1[(

minus= +wait N

timeout

45

bull Most waits were for nothing ndash latch already was free

bull Latch utilization can not be high

bull Unnecessary spins provoke CPU thrashing

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682

92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE

sqlplus nolog demo_05sql 45

RDTEX | Latch internals | RUOUG Seminar

kslgetl(0x50006318 1)

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt kslges(0x50006318 ) -wait latch get

-gt skgslsgts(0x50006318 ) -spin latch get

-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT

-gt kskthbwt(0x0)

-gt kslwlmod() - set up Wait List

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt skgpwwait -sleep latch getsemop(11 17-10 1)

983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140

46

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782

RELIABLE LATCH WAITS

bull Hidden latch wait revolution that we missed Latch relies on waitposting

bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout

bull Exclusive latch spins 20000 cycles by default

bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock

bull Latches assigned to non-default class wait until timeout

bull Version 90 was transient and used mix of backoff and wait posting

This is why _LATCH_WAIT_POSTING parameter disappeared in9i

bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin

RDTEX | Latch internals | RUOUG Seminar

47

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882

IMAGINE THE TIME MICROSCOPE

| a a s | a

Reality One million zoomed time

1 us 1 sec

1 ms 17 min1 sec 115 days

Light speed (300000 kmsec) Sonic speed (300 msec)

CPU tick (2 GHz) 00005 sec

Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec

Max spin time for shared latch (~220005 ticks) ~10 sec

Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec

Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec

OS context switch time (10us-1ms) 10sec-17min

48

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982

LATCH WAITS UNDER TIME MICROSCOPE

Old 8i latch algorithmSpin for latch during 5 seconds

Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min

hellipSpin again for 5 seconds

Sleep again for 23 days (2 sec) hellip

New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)

If spin not succeeded go to sleep until post

According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)

| |

49

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082

_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES

bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and

_LATCH_CLASS_0 parameters

bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000

bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter

bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch

50

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182

TIMED LATCH WAIT POSTING

bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel

bull This can lead to instance hang ndash process will never be woken up

bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout

bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem

bull Generic AIX ports always use timed latch wait posting

bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux

51

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282

FINE GRAIN LATCH TUNING

bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting

bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7

bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo

microseconds using pollsys() (not semtimedop()) system call

ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times

Loop up to ldquoSpinsrdquo cycles

Yield CPU using yield() (or sched_yield())

ndash Sleep for ldquoSleepXrdquo usecs

ndash Then spin again hellip

52

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382

TRACES OF NONDEFAULT CLASS LATCH

983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152

991270

9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000

991270 E983158983141983150983156 10046 983156983154983137983139983141

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482

PERFORMANCE OF LATCH CLASSES

Row cache objects latch contention testcase on ODA (24 SMT)54

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582

IS THE _SPIN_COUNT REALLY STATIC

bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch

waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used

ldquocache buffers chainrdquo latch contention

bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass

(_LATCH_CLASS_X) By default ndash 20000

bull Shared latch spin can be tuned by _SPIN_COUNT value By

default ndash 2000 However this is not trivial

55

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682

SHARED LATCH BEHAVE LIKE ENQUEUE

bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80

bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests

bull This blocking achieved by special 0x40000000 bit in the latch

value This bit is a flag that some incompatible latch acquisitionis in progress

bull Queued processes will be woken up one by one on latch release

bull X mode latch gets effectively serialize the shared latch

bull See my blog for description of experiments

56

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782

TRACE OF SHARED LATCH WAIT

Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the

latch wait list and then spins again

983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141

983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156

983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983147983155983148983159983148983149983151983140()

983147983155983148983155983147983143983155(098316050009BAC )

983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140

983155983141983149983151983152(27)

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882

SHARED LATCH ACQUISITION

bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned

bull X mode shared latch get spins by default up to 4000 cycles

bull S mode does not spin at all (or spins in unknown way)

S mode get X mode get

Held in S mode Compatible 2_SPIN_COUNT

Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT

58

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982

LATCH RELEASE

bull Free the latch ndash kslfre(laddr)

bull Oracle process releases the latch nonatomically

bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process

bull This requires less bus invalidation and ensures propagation of latch

release to other local cachesbull Not fair policy Spinners on the local CPU board have the

preference

bull Then process posts the first process in the list of waiters

59

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082

LATCH CONTENTION

RDTEX | Latch internals | RUOUG Seminar60

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182

LATCH CONTENTION

bull Occurs when a latch is required by several processes at thesame time

bull Mostly due to bugs or abnormal application behavior

bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp

bull To diagnose the latch contention we need to investigate thelatch statistics [8]

61

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282

DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS

Latch requests arrival rate

Immediate gets efficiency

Latch sleeps ratio

Latch wait time per second

Latch spin efficiencymisses

getsspin∆

∆= _σ

misses

sleepsK

∆=

time

gets

∆=λ

gets

misses

∆= ρ

time

timewait W

∆=

_

Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics

62

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382

SAMPLED AVERAGES

By sampling of x$ksupr we can estimate

ndash Length of latch wait queue

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS

ndash Number of spinning processes in 10g

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS

Sampling of v$lathholder estimates latch utilization

Using DTrace we can measure

ndash Latch acquisition and holding times

ndash Actual spin count

w L

s N

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482

DERIVED LATCH STATISTICS

Latch utilization (PASTA)

Average holding time

Length of latch wait list

Recurrent sleeps ratio

Latch acquisition time

time

timeholdinglatchU

∆=asymp

__ ρ

Re_100

___

questsGet

TimeSnap MissGet Pct S

lowast==

λ

W L =

κ

κ σ 1minus+

)(1

W N T saq += minusλ

64

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 45: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4582

POTENTIAL PROBLEMS WITH EXPONENTIAL BACKOFF

bull 001-001-001-003-003-007-007-015-023-039-039-071-071-135-135-20-20-20-20sec

RDTEX | Latch internals | RUOUG Seminar

12 ]2 )1[(

minus= +wait N

timeout

45

bull Most waits were for nothing ndash latch already was free

bull Latch utilization can not be high

bull Unnecessary spins provoke CPU thrashing

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682

92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE

sqlplus nolog demo_05sql 45

RDTEX | Latch internals | RUOUG Seminar

kslgetl(0x50006318 1)

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt kslges(0x50006318 ) -wait latch get

-gt skgslsgts(0x50006318 ) -spin latch get

-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT

-gt kskthbwt(0x0)

-gt kslwlmod() - set up Wait List

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt skgpwwait -sleep latch getsemop(11 17-10 1)

983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140

46

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782

RELIABLE LATCH WAITS

bull Hidden latch wait revolution that we missed Latch relies on waitposting

bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout

bull Exclusive latch spins 20000 cycles by default

bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock

bull Latches assigned to non-default class wait until timeout

bull Version 90 was transient and used mix of backoff and wait posting

This is why _LATCH_WAIT_POSTING parameter disappeared in9i

bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin

RDTEX | Latch internals | RUOUG Seminar

47

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882

IMAGINE THE TIME MICROSCOPE

| a a s | a

Reality One million zoomed time

1 us 1 sec

1 ms 17 min1 sec 115 days

Light speed (300000 kmsec) Sonic speed (300 msec)

CPU tick (2 GHz) 00005 sec

Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec

Max spin time for shared latch (~220005 ticks) ~10 sec

Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec

Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec

OS context switch time (10us-1ms) 10sec-17min

48

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982

LATCH WAITS UNDER TIME MICROSCOPE

Old 8i latch algorithmSpin for latch during 5 seconds

Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min

hellipSpin again for 5 seconds

Sleep again for 23 days (2 sec) hellip

New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)

If spin not succeeded go to sleep until post

According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)

| |

49

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082

_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES

bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and

_LATCH_CLASS_0 parameters

bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000

bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter

bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch

50

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182

TIMED LATCH WAIT POSTING

bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel

bull This can lead to instance hang ndash process will never be woken up

bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout

bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem

bull Generic AIX ports always use timed latch wait posting

bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux

51

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282

FINE GRAIN LATCH TUNING

bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting

bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7

bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo

microseconds using pollsys() (not semtimedop()) system call

ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times

Loop up to ldquoSpinsrdquo cycles

Yield CPU using yield() (or sched_yield())

ndash Sleep for ldquoSleepXrdquo usecs

ndash Then spin again hellip

52

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382

TRACES OF NONDEFAULT CLASS LATCH

983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152

991270

9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000

991270 E983158983141983150983156 10046 983156983154983137983139983141

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482

PERFORMANCE OF LATCH CLASSES

Row cache objects latch contention testcase on ODA (24 SMT)54

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582

IS THE _SPIN_COUNT REALLY STATIC

bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch

waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used

ldquocache buffers chainrdquo latch contention

bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass

(_LATCH_CLASS_X) By default ndash 20000

bull Shared latch spin can be tuned by _SPIN_COUNT value By

default ndash 2000 However this is not trivial

55

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682

SHARED LATCH BEHAVE LIKE ENQUEUE

bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80

bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests

bull This blocking achieved by special 0x40000000 bit in the latch

value This bit is a flag that some incompatible latch acquisitionis in progress

bull Queued processes will be woken up one by one on latch release

bull X mode latch gets effectively serialize the shared latch

bull See my blog for description of experiments

56

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782

TRACE OF SHARED LATCH WAIT

Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the

latch wait list and then spins again

983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141

983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156

983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983147983155983148983159983148983149983151983140()

983147983155983148983155983147983143983155(098316050009BAC )

983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140

983155983141983149983151983152(27)

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882

SHARED LATCH ACQUISITION

bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned

bull X mode shared latch get spins by default up to 4000 cycles

bull S mode does not spin at all (or spins in unknown way)

S mode get X mode get

Held in S mode Compatible 2_SPIN_COUNT

Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT

58

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982

LATCH RELEASE

bull Free the latch ndash kslfre(laddr)

bull Oracle process releases the latch nonatomically

bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process

bull This requires less bus invalidation and ensures propagation of latch

release to other local cachesbull Not fair policy Spinners on the local CPU board have the

preference

bull Then process posts the first process in the list of waiters

59

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082

LATCH CONTENTION

RDTEX | Latch internals | RUOUG Seminar60

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182

LATCH CONTENTION

bull Occurs when a latch is required by several processes at thesame time

bull Mostly due to bugs or abnormal application behavior

bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp

bull To diagnose the latch contention we need to investigate thelatch statistics [8]

61

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282

DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS

Latch requests arrival rate

Immediate gets efficiency

Latch sleeps ratio

Latch wait time per second

Latch spin efficiencymisses

getsspin∆

∆= _σ

misses

sleepsK

∆=

time

gets

∆=λ

gets

misses

∆= ρ

time

timewait W

∆=

_

Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics

62

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382

SAMPLED AVERAGES

By sampling of x$ksupr we can estimate

ndash Length of latch wait queue

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS

ndash Number of spinning processes in 10g

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS

Sampling of v$lathholder estimates latch utilization

Using DTrace we can measure

ndash Latch acquisition and holding times

ndash Actual spin count

w L

s N

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482

DERIVED LATCH STATISTICS

Latch utilization (PASTA)

Average holding time

Length of latch wait list

Recurrent sleeps ratio

Latch acquisition time

time

timeholdinglatchU

∆=asymp

__ ρ

Re_100

___

questsGet

TimeSnap MissGet Pct S

lowast==

λ

W L =

κ

κ σ 1minus+

)(1

W N T saq += minusλ

64

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 46: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4682

92-11G EXCLUSIVE LATCH SPIN FLOW USING DTRACE

sqlplus nolog demo_05sql 45

RDTEX | Latch internals | RUOUG Seminar

kslgetl(0x50006318 1)

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt kslges(0x50006318 ) -wait latch get

-gt skgslsgts(0x50006318 ) -spin latch get

-gtsskgslspin(0x50006318) - repeated 20000 cycles = 10_SPIN_COUNT

-gt kskthbwt(0x0)

-gt kslwlmod() - set up Wait List

-gt sskgslgf(0x50006318)= 0 -immediate latch get

-gt skgpwwait -sleep latch getsemop(11 17-10 1)

983123983141983149983151983152 991251 983145983150983142983145983150983145983156983141 983159983137983145983156 983157983150983156983145983148 983152983151983155983156983141983140

46

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782

RELIABLE LATCH WAITS

bull Hidden latch wait revolution that we missed Latch relies on waitposting

bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout

bull Exclusive latch spins 20000 cycles by default

bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock

bull Latches assigned to non-default class wait until timeout

bull Version 90 was transient and used mix of backoff and wait posting

This is why _LATCH_WAIT_POSTING parameter disappeared in9i

bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin

RDTEX | Latch internals | RUOUG Seminar

47

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882

IMAGINE THE TIME MICROSCOPE

| a a s | a

Reality One million zoomed time

1 us 1 sec

1 ms 17 min1 sec 115 days

Light speed (300000 kmsec) Sonic speed (300 msec)

CPU tick (2 GHz) 00005 sec

Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec

Max spin time for shared latch (~220005 ticks) ~10 sec

Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec

Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec

OS context switch time (10us-1ms) 10sec-17min

48

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982

LATCH WAITS UNDER TIME MICROSCOPE

Old 8i latch algorithmSpin for latch during 5 seconds

Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min

hellipSpin again for 5 seconds

Sleep again for 23 days (2 sec) hellip

New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)

If spin not succeeded go to sleep until post

According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)

| |

49

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082

_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES

bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and

_LATCH_CLASS_0 parameters

bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000

bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter

bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch

50

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182

TIMED LATCH WAIT POSTING

bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel

bull This can lead to instance hang ndash process will never be woken up

bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout

bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem

bull Generic AIX ports always use timed latch wait posting

bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux

51

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282

FINE GRAIN LATCH TUNING

bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting

bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7

bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo

microseconds using pollsys() (not semtimedop()) system call

ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times

Loop up to ldquoSpinsrdquo cycles

Yield CPU using yield() (or sched_yield())

ndash Sleep for ldquoSleepXrdquo usecs

ndash Then spin again hellip

52

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382

TRACES OF NONDEFAULT CLASS LATCH

983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152

991270

9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000

991270 E983158983141983150983156 10046 983156983154983137983139983141

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482

PERFORMANCE OF LATCH CLASSES

Row cache objects latch contention testcase on ODA (24 SMT)54

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582

IS THE _SPIN_COUNT REALLY STATIC

bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch

waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used

ldquocache buffers chainrdquo latch contention

bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass

(_LATCH_CLASS_X) By default ndash 20000

bull Shared latch spin can be tuned by _SPIN_COUNT value By

default ndash 2000 However this is not trivial

55

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682

SHARED LATCH BEHAVE LIKE ENQUEUE

bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80

bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests

bull This blocking achieved by special 0x40000000 bit in the latch

value This bit is a flag that some incompatible latch acquisitionis in progress

bull Queued processes will be woken up one by one on latch release

bull X mode latch gets effectively serialize the shared latch

bull See my blog for description of experiments

56

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782

TRACE OF SHARED LATCH WAIT

Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the

latch wait list and then spins again

983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141

983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156

983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983147983155983148983159983148983149983151983140()

983147983155983148983155983147983143983155(098316050009BAC )

983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140

983155983141983149983151983152(27)

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882

SHARED LATCH ACQUISITION

bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned

bull X mode shared latch get spins by default up to 4000 cycles

bull S mode does not spin at all (or spins in unknown way)

S mode get X mode get

Held in S mode Compatible 2_SPIN_COUNT

Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT

58

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982

LATCH RELEASE

bull Free the latch ndash kslfre(laddr)

bull Oracle process releases the latch nonatomically

bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process

bull This requires less bus invalidation and ensures propagation of latch

release to other local cachesbull Not fair policy Spinners on the local CPU board have the

preference

bull Then process posts the first process in the list of waiters

59

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082

LATCH CONTENTION

RDTEX | Latch internals | RUOUG Seminar60

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182

LATCH CONTENTION

bull Occurs when a latch is required by several processes at thesame time

bull Mostly due to bugs or abnormal application behavior

bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp

bull To diagnose the latch contention we need to investigate thelatch statistics [8]

61

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282

DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS

Latch requests arrival rate

Immediate gets efficiency

Latch sleeps ratio

Latch wait time per second

Latch spin efficiencymisses

getsspin∆

∆= _σ

misses

sleepsK

∆=

time

gets

∆=λ

gets

misses

∆= ρ

time

timewait W

∆=

_

Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics

62

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382

SAMPLED AVERAGES

By sampling of x$ksupr we can estimate

ndash Length of latch wait queue

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS

ndash Number of spinning processes in 10g

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS

Sampling of v$lathholder estimates latch utilization

Using DTrace we can measure

ndash Latch acquisition and holding times

ndash Actual spin count

w L

s N

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482

DERIVED LATCH STATISTICS

Latch utilization (PASTA)

Average holding time

Length of latch wait list

Recurrent sleeps ratio

Latch acquisition time

time

timeholdinglatchU

∆=asymp

__ ρ

Re_100

___

questsGet

TimeSnap MissGet Pct S

lowast==

λ

W L =

κ

κ σ 1minus+

)(1

W N T saq += minusλ

64

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 47: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4782

RELIABLE LATCH WAITS

bull Hidden latch wait revolution that we missed Latch relies on waitposting

bull In Oracle 92-112 all the latches in default class 0 use latch waitposting Process waits for the latch without any timeout

bull Exclusive latch spins 20000 cycles by default

bull Only first read is atomic further reads poll the CPU cache Latch isTTS spinlock

bull Latches assigned to non-default class wait until timeout

bull Version 90 was transient and used mix of backoff and wait posting

This is why _LATCH_WAIT_POSTING parameter disappeared in9i

bull _LATCH_CLASSES and _LATCH_CLASS_X parametersdetermine the latch wait and spin

RDTEX | Latch internals | RUOUG Seminar

47

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882

IMAGINE THE TIME MICROSCOPE

| a a s | a

Reality One million zoomed time

1 us 1 sec

1 ms 17 min1 sec 115 days

Light speed (300000 kmsec) Sonic speed (300 msec)

CPU tick (2 GHz) 00005 sec

Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec

Max spin time for shared latch (~220005 ticks) ~10 sec

Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec

Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec

OS context switch time (10us-1ms) 10sec-17min

48

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982

LATCH WAITS UNDER TIME MICROSCOPE

Old 8i latch algorithmSpin for latch during 5 seconds

Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min

hellipSpin again for 5 seconds

Sleep again for 23 days (2 sec) hellip

New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)

If spin not succeeded go to sleep until post

According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)

| |

49

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082

_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES

bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and

_LATCH_CLASS_0 parameters

bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000

bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter

bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch

50

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182

TIMED LATCH WAIT POSTING

bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel

bull This can lead to instance hang ndash process will never be woken up

bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout

bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem

bull Generic AIX ports always use timed latch wait posting

bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux

51

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282

FINE GRAIN LATCH TUNING

bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting

bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7

bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo

microseconds using pollsys() (not semtimedop()) system call

ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times

Loop up to ldquoSpinsrdquo cycles

Yield CPU using yield() (or sched_yield())

ndash Sleep for ldquoSleepXrdquo usecs

ndash Then spin again hellip

52

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382

TRACES OF NONDEFAULT CLASS LATCH

983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152

991270

9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000

991270 E983158983141983150983156 10046 983156983154983137983139983141

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482

PERFORMANCE OF LATCH CLASSES

Row cache objects latch contention testcase on ODA (24 SMT)54

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582

IS THE _SPIN_COUNT REALLY STATIC

bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch

waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used

ldquocache buffers chainrdquo latch contention

bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass

(_LATCH_CLASS_X) By default ndash 20000

bull Shared latch spin can be tuned by _SPIN_COUNT value By

default ndash 2000 However this is not trivial

55

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682

SHARED LATCH BEHAVE LIKE ENQUEUE

bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80

bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests

bull This blocking achieved by special 0x40000000 bit in the latch

value This bit is a flag that some incompatible latch acquisitionis in progress

bull Queued processes will be woken up one by one on latch release

bull X mode latch gets effectively serialize the shared latch

bull See my blog for description of experiments

56

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782

TRACE OF SHARED LATCH WAIT

Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the

latch wait list and then spins again

983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141

983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156

983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983147983155983148983159983148983149983151983140()

983147983155983148983155983147983143983155(098316050009BAC )

983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140

983155983141983149983151983152(27)

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882

SHARED LATCH ACQUISITION

bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned

bull X mode shared latch get spins by default up to 4000 cycles

bull S mode does not spin at all (or spins in unknown way)

S mode get X mode get

Held in S mode Compatible 2_SPIN_COUNT

Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT

58

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982

LATCH RELEASE

bull Free the latch ndash kslfre(laddr)

bull Oracle process releases the latch nonatomically

bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process

bull This requires less bus invalidation and ensures propagation of latch

release to other local cachesbull Not fair policy Spinners on the local CPU board have the

preference

bull Then process posts the first process in the list of waiters

59

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082

LATCH CONTENTION

RDTEX | Latch internals | RUOUG Seminar60

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182

LATCH CONTENTION

bull Occurs when a latch is required by several processes at thesame time

bull Mostly due to bugs or abnormal application behavior

bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp

bull To diagnose the latch contention we need to investigate thelatch statistics [8]

61

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282

DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS

Latch requests arrival rate

Immediate gets efficiency

Latch sleeps ratio

Latch wait time per second

Latch spin efficiencymisses

getsspin∆

∆= _σ

misses

sleepsK

∆=

time

gets

∆=λ

gets

misses

∆= ρ

time

timewait W

∆=

_

Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics

62

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382

SAMPLED AVERAGES

By sampling of x$ksupr we can estimate

ndash Length of latch wait queue

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS

ndash Number of spinning processes in 10g

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS

Sampling of v$lathholder estimates latch utilization

Using DTrace we can measure

ndash Latch acquisition and holding times

ndash Actual spin count

w L

s N

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482

DERIVED LATCH STATISTICS

Latch utilization (PASTA)

Average holding time

Length of latch wait list

Recurrent sleeps ratio

Latch acquisition time

time

timeholdinglatchU

∆=asymp

__ ρ

Re_100

___

questsGet

TimeSnap MissGet Pct S

lowast==

λ

W L =

κ

κ σ 1minus+

)(1

W N T saq += minusλ

64

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 48: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4882

IMAGINE THE TIME MICROSCOPE

| a a s | a

Reality One million zoomed time

1 us 1 sec

1 ms 17 min1 sec 115 days

Light speed (300000 kmsec) Sonic speed (300 msec)

CPU tick (2 GHz) 00005 sec

Avg ldquoCache buffers chainsrdquo latch holding time ~ 1-2 sec

Max spin time for shared latch (~220005 ticks) ~10 sec

Avg ldquoLibrary Cacherdquo latch holding time (10-20us) ~ 10-20 sec

Max spin time for exclusive latch (~200005 ticks) ~50 secInterval between CBC latch gets ~ 10 sec

OS context switch time (10us-1ms) 10sec-17min

48

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982

LATCH WAITS UNDER TIME MICROSCOPE

Old 8i latch algorithmSpin for latch during 5 seconds

Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min

hellipSpin again for 5 seconds

Sleep again for 23 days (2 sec) hellip

New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)

If spin not succeeded go to sleep until post

According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)

| |

49

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082

_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES

bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and

_LATCH_CLASS_0 parameters

bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000

bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter

bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch

50

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182

TIMED LATCH WAIT POSTING

bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel

bull This can lead to instance hang ndash process will never be woken up

bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout

bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem

bull Generic AIX ports always use timed latch wait posting

bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux

51

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282

FINE GRAIN LATCH TUNING

bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting

bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7

bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo

microseconds using pollsys() (not semtimedop()) system call

ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times

Loop up to ldquoSpinsrdquo cycles

Yield CPU using yield() (or sched_yield())

ndash Sleep for ldquoSleepXrdquo usecs

ndash Then spin again hellip

52

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382

TRACES OF NONDEFAULT CLASS LATCH

983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152

991270

9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000

991270 E983158983141983150983156 10046 983156983154983137983139983141

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482

PERFORMANCE OF LATCH CLASSES

Row cache objects latch contention testcase on ODA (24 SMT)54

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582

IS THE _SPIN_COUNT REALLY STATIC

bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch

waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used

ldquocache buffers chainrdquo latch contention

bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass

(_LATCH_CLASS_X) By default ndash 20000

bull Shared latch spin can be tuned by _SPIN_COUNT value By

default ndash 2000 However this is not trivial

55

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682

SHARED LATCH BEHAVE LIKE ENQUEUE

bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80

bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests

bull This blocking achieved by special 0x40000000 bit in the latch

value This bit is a flag that some incompatible latch acquisitionis in progress

bull Queued processes will be woken up one by one on latch release

bull X mode latch gets effectively serialize the shared latch

bull See my blog for description of experiments

56

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782

TRACE OF SHARED LATCH WAIT

Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the

latch wait list and then spins again

983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141

983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156

983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983147983155983148983159983148983149983151983140()

983147983155983148983155983147983143983155(098316050009BAC )

983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140

983155983141983149983151983152(27)

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882

SHARED LATCH ACQUISITION

bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned

bull X mode shared latch get spins by default up to 4000 cycles

bull S mode does not spin at all (or spins in unknown way)

S mode get X mode get

Held in S mode Compatible 2_SPIN_COUNT

Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT

58

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982

LATCH RELEASE

bull Free the latch ndash kslfre(laddr)

bull Oracle process releases the latch nonatomically

bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process

bull This requires less bus invalidation and ensures propagation of latch

release to other local cachesbull Not fair policy Spinners on the local CPU board have the

preference

bull Then process posts the first process in the list of waiters

59

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082

LATCH CONTENTION

RDTEX | Latch internals | RUOUG Seminar60

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182

LATCH CONTENTION

bull Occurs when a latch is required by several processes at thesame time

bull Mostly due to bugs or abnormal application behavior

bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp

bull To diagnose the latch contention we need to investigate thelatch statistics [8]

61

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282

DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS

Latch requests arrival rate

Immediate gets efficiency

Latch sleeps ratio

Latch wait time per second

Latch spin efficiencymisses

getsspin∆

∆= _σ

misses

sleepsK

∆=

time

gets

∆=λ

gets

misses

∆= ρ

time

timewait W

∆=

_

Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics

62

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382

SAMPLED AVERAGES

By sampling of x$ksupr we can estimate

ndash Length of latch wait queue

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS

ndash Number of spinning processes in 10g

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS

Sampling of v$lathholder estimates latch utilization

Using DTrace we can measure

ndash Latch acquisition and holding times

ndash Actual spin count

w L

s N

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482

DERIVED LATCH STATISTICS

Latch utilization (PASTA)

Average holding time

Length of latch wait list

Recurrent sleeps ratio

Latch acquisition time

time

timeholdinglatchU

∆=asymp

__ ρ

Re_100

___

questsGet

TimeSnap MissGet Pct S

lowast==

λ

W L =

κ

κ σ 1minus+

)(1

W N T saq += minusλ

64

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 49: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 4982

LATCH WAITS UNDER TIME MICROSCOPE

Old 8i latch algorithmSpin for latch during 5 seconds

Go sleep for 2 hours 46 min in hope that congestion will dissolveSpin again during 5 secondsSleep again for 2 hours 46 min

hellipSpin again for 5 seconds

Sleep again for 23 days (2 sec) hellip

New 92 latch algorithmSpin for shared latch during 10 seconds (50 for exclusive latch)

If spin not succeeded go to sleep until post

According to v$event_histogram the majority of latch waits now takeless then 17 min (1ms)

| |

49

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082

_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES

bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and

_LATCH_CLASS_0 parameters

bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000

bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter

bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch

50

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182

TIMED LATCH WAIT POSTING

bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel

bull This can lead to instance hang ndash process will never be woken up

bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout

bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem

bull Generic AIX ports always use timed latch wait posting

bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux

51

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282

FINE GRAIN LATCH TUNING

bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting

bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7

bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo

microseconds using pollsys() (not semtimedop()) system call

ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times

Loop up to ldquoSpinsrdquo cycles

Yield CPU using yield() (or sched_yield())

ndash Sleep for ldquoSleepXrdquo usecs

ndash Then spin again hellip

52

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382

TRACES OF NONDEFAULT CLASS LATCH

983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152

991270

9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000

991270 E983158983141983150983156 10046 983156983154983137983139983141

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482

PERFORMANCE OF LATCH CLASSES

Row cache objects latch contention testcase on ODA (24 SMT)54

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582

IS THE _SPIN_COUNT REALLY STATIC

bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch

waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used

ldquocache buffers chainrdquo latch contention

bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass

(_LATCH_CLASS_X) By default ndash 20000

bull Shared latch spin can be tuned by _SPIN_COUNT value By

default ndash 2000 However this is not trivial

55

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682

SHARED LATCH BEHAVE LIKE ENQUEUE

bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80

bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests

bull This blocking achieved by special 0x40000000 bit in the latch

value This bit is a flag that some incompatible latch acquisitionis in progress

bull Queued processes will be woken up one by one on latch release

bull X mode latch gets effectively serialize the shared latch

bull See my blog for description of experiments

56

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782

TRACE OF SHARED LATCH WAIT

Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the

latch wait list and then spins again

983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141

983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156

983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983147983155983148983159983148983149983151983140()

983147983155983148983155983147983143983155(098316050009BAC )

983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140

983155983141983149983151983152(27)

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882

SHARED LATCH ACQUISITION

bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned

bull X mode shared latch get spins by default up to 4000 cycles

bull S mode does not spin at all (or spins in unknown way)

S mode get X mode get

Held in S mode Compatible 2_SPIN_COUNT

Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT

58

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982

LATCH RELEASE

bull Free the latch ndash kslfre(laddr)

bull Oracle process releases the latch nonatomically

bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process

bull This requires less bus invalidation and ensures propagation of latch

release to other local cachesbull Not fair policy Spinners on the local CPU board have the

preference

bull Then process posts the first process in the list of waiters

59

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082

LATCH CONTENTION

RDTEX | Latch internals | RUOUG Seminar60

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182

LATCH CONTENTION

bull Occurs when a latch is required by several processes at thesame time

bull Mostly due to bugs or abnormal application behavior

bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp

bull To diagnose the latch contention we need to investigate thelatch statistics [8]

61

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282

DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS

Latch requests arrival rate

Immediate gets efficiency

Latch sleeps ratio

Latch wait time per second

Latch spin efficiencymisses

getsspin∆

∆= _σ

misses

sleepsK

∆=

time

gets

∆=λ

gets

misses

∆= ρ

time

timewait W

∆=

_

Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics

62

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382

SAMPLED AVERAGES

By sampling of x$ksupr we can estimate

ndash Length of latch wait queue

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS

ndash Number of spinning processes in 10g

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS

Sampling of v$lathholder estimates latch utilization

Using DTrace we can measure

ndash Latch acquisition and holding times

ndash Actual spin count

w L

s N

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482

DERIVED LATCH STATISTICS

Latch utilization (PASTA)

Average holding time

Length of latch wait list

Recurrent sleeps ratio

Latch acquisition time

time

timeholdinglatchU

∆=asymp

__ ρ

Re_100

___

questsGet

TimeSnap MissGet Pct S

lowast==

λ

W L =

κ

κ σ 1minus+

)(1

W N T saq += minusλ

64

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 50: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5082

_SPIN_COUNT IS EFFECTIVELY STATICFOR EXCLUSIVE LATCHES

bull Oracle process spins for exclusive latch up to x$ksllclass[0]spincycles This number depends on _SPIN_COUNT and

_LATCH_CLASS_0 parameters

bull If _SPIN_COUNT was not set during instance startup ndash process willspin up to 20000 cycles The default value for _SPIN_COUNT isonly 2000

bull If instance started with _SPIN_COUNT parameter Oracle setx$ksllclassspin to _SPIN_COUNT if not otherwise specified by _LATCH_CLASS_0 parameter

bull The _SPIN_COUNT parameter can be changed dynamically butthis change will not have any effect for exclusive latch

50

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182

TIMED LATCH WAIT POSTING

bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel

bull This can lead to instance hang ndash process will never be woken up

bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout

bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem

bull Generic AIX ports always use timed latch wait posting

bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux

51

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282

FINE GRAIN LATCH TUNING

bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting

bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7

bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo

microseconds using pollsys() (not semtimedop()) system call

ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times

Loop up to ldquoSpinsrdquo cycles

Yield CPU using yield() (or sched_yield())

ndash Sleep for ldquoSleepXrdquo usecs

ndash Then spin again hellip

52

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382

TRACES OF NONDEFAULT CLASS LATCH

983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152

991270

9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000

991270 E983158983141983150983156 10046 983156983154983137983139983141

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482

PERFORMANCE OF LATCH CLASSES

Row cache objects latch contention testcase on ODA (24 SMT)54

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582

IS THE _SPIN_COUNT REALLY STATIC

bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch

waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used

ldquocache buffers chainrdquo latch contention

bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass

(_LATCH_CLASS_X) By default ndash 20000

bull Shared latch spin can be tuned by _SPIN_COUNT value By

default ndash 2000 However this is not trivial

55

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682

SHARED LATCH BEHAVE LIKE ENQUEUE

bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80

bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests

bull This blocking achieved by special 0x40000000 bit in the latch

value This bit is a flag that some incompatible latch acquisitionis in progress

bull Queued processes will be woken up one by one on latch release

bull X mode latch gets effectively serialize the shared latch

bull See my blog for description of experiments

56

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782

TRACE OF SHARED LATCH WAIT

Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the

latch wait list and then spins again

983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141

983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156

983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983147983155983148983159983148983149983151983140()

983147983155983148983155983147983143983155(098316050009BAC )

983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140

983155983141983149983151983152(27)

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882

SHARED LATCH ACQUISITION

bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned

bull X mode shared latch get spins by default up to 4000 cycles

bull S mode does not spin at all (or spins in unknown way)

S mode get X mode get

Held in S mode Compatible 2_SPIN_COUNT

Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT

58

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982

LATCH RELEASE

bull Free the latch ndash kslfre(laddr)

bull Oracle process releases the latch nonatomically

bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process

bull This requires less bus invalidation and ensures propagation of latch

release to other local cachesbull Not fair policy Spinners on the local CPU board have the

preference

bull Then process posts the first process in the list of waiters

59

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082

LATCH CONTENTION

RDTEX | Latch internals | RUOUG Seminar60

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182

LATCH CONTENTION

bull Occurs when a latch is required by several processes at thesame time

bull Mostly due to bugs or abnormal application behavior

bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp

bull To diagnose the latch contention we need to investigate thelatch statistics [8]

61

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282

DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS

Latch requests arrival rate

Immediate gets efficiency

Latch sleeps ratio

Latch wait time per second

Latch spin efficiencymisses

getsspin∆

∆= _σ

misses

sleepsK

∆=

time

gets

∆=λ

gets

misses

∆= ρ

time

timewait W

∆=

_

Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics

62

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382

SAMPLED AVERAGES

By sampling of x$ksupr we can estimate

ndash Length of latch wait queue

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS

ndash Number of spinning processes in 10g

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS

Sampling of v$lathholder estimates latch utilization

Using DTrace we can measure

ndash Latch acquisition and holding times

ndash Actual spin count

w L

s N

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482

DERIVED LATCH STATISTICS

Latch utilization (PASTA)

Average holding time

Length of latch wait list

Recurrent sleeps ratio

Latch acquisition time

time

timeholdinglatchU

∆=asymp

__ ρ

Re_100

___

questsGet

TimeSnap MissGet Pct S

lowast==

λ

W L =

κ

κ σ 1minus+

)(1

W N T saq += minusλ

64

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 51: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5182

TIMED LATCH WAIT POSTING

bull If wakeup post is lost in OS waiters will sleep infinitely Commonproblem in earlier 269 Linux kernel

bull This can lead to instance hang ndash process will never be woken up

bull _ENABLE_RELIABLE_LATCH_WAITS = FALSE changes semop()system call to semtimedop() with 03 sec timeout

bull Introduced in 9206 for Bug 3632400 LMS may hang in kslges()waiting for a post Increased robustness to OS posting problem

bull Generic AIX ports always use timed latch wait posting

bull My testcases did not revealed any measurable performanceoverhead of timed latch wait posting in Solaris and Linux

51

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282

FINE GRAIN LATCH TUNING

bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting

bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7

bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo

microseconds using pollsys() (not semtimedop()) system call

ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times

Loop up to ldquoSpinsrdquo cycles

Yield CPU using yield() (or sched_yield())

ndash Sleep for ldquoSleepXrdquo usecs

ndash Then spin again hellip

52

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382

TRACES OF NONDEFAULT CLASS LATCH

983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152

991270

9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000

991270 E983158983141983150983156 10046 983156983154983137983139983141

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482

PERFORMANCE OF LATCH CLASSES

Row cache objects latch contention testcase on ODA (24 SMT)54

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582

IS THE _SPIN_COUNT REALLY STATIC

bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch

waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used

ldquocache buffers chainrdquo latch contention

bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass

(_LATCH_CLASS_X) By default ndash 20000

bull Shared latch spin can be tuned by _SPIN_COUNT value By

default ndash 2000 However this is not trivial

55

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682

SHARED LATCH BEHAVE LIKE ENQUEUE

bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80

bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests

bull This blocking achieved by special 0x40000000 bit in the latch

value This bit is a flag that some incompatible latch acquisitionis in progress

bull Queued processes will be woken up one by one on latch release

bull X mode latch gets effectively serialize the shared latch

bull See my blog for description of experiments

56

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782

TRACE OF SHARED LATCH WAIT

Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the

latch wait list and then spins again

983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141

983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156

983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983147983155983148983159983148983149983151983140()

983147983155983148983155983147983143983155(098316050009BAC )

983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140

983155983141983149983151983152(27)

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882

SHARED LATCH ACQUISITION

bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned

bull X mode shared latch get spins by default up to 4000 cycles

bull S mode does not spin at all (or spins in unknown way)

S mode get X mode get

Held in S mode Compatible 2_SPIN_COUNT

Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT

58

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982

LATCH RELEASE

bull Free the latch ndash kslfre(laddr)

bull Oracle process releases the latch nonatomically

bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process

bull This requires less bus invalidation and ensures propagation of latch

release to other local cachesbull Not fair policy Spinners on the local CPU board have the

preference

bull Then process posts the first process in the list of waiters

59

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082

LATCH CONTENTION

RDTEX | Latch internals | RUOUG Seminar60

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182

LATCH CONTENTION

bull Occurs when a latch is required by several processes at thesame time

bull Mostly due to bugs or abnormal application behavior

bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp

bull To diagnose the latch contention we need to investigate thelatch statistics [8]

61

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282

DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS

Latch requests arrival rate

Immediate gets efficiency

Latch sleeps ratio

Latch wait time per second

Latch spin efficiencymisses

getsspin∆

∆= _σ

misses

sleepsK

∆=

time

gets

∆=λ

gets

misses

∆= ρ

time

timewait W

∆=

_

Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics

62

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382

SAMPLED AVERAGES

By sampling of x$ksupr we can estimate

ndash Length of latch wait queue

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS

ndash Number of spinning processes in 10g

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS

Sampling of v$lathholder estimates latch utilization

Using DTrace we can measure

ndash Latch acquisition and holding times

ndash Actual spin count

w L

s N

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482

DERIVED LATCH STATISTICS

Latch utilization (PASTA)

Average holding time

Length of latch wait list

Recurrent sleeps ratio

Latch acquisition time

time

timeholdinglatchU

∆=asymp

__ ρ

Re_100

___

questsGet

TimeSnap MissGet Pct S

lowast==

λ

W L =

κ

κ σ 1minus+

)(1

W N T saq += minusλ

64

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 52: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5282

FINE GRAIN LATCH TUNING

bull Latch can be assigned to one of eight classes having different spin andwait policies Standard class 0 latch use wait posting

bull _latch_class_X = ldquoSpin Yield Waittime Sleep0 Sleep1 hellip Sleep7

bull Nonstandard class latch loops upto ldquoSpinrdquo cycles then yields CPU This isrepeated ldquoYieldrdquo times Then the process sleeps for ldquoSleepXrdquo

microseconds using pollsys() (not semtimedop()) system call

ndash If ldquoYieldrdquo =0 repeat ldquoYieldrdquo times

Loop up to ldquoSpinsrdquo cycles

Yield CPU using yield() (or sched_yield())

ndash Sleep for ldquoSleepXrdquo usecs

ndash Then spin again hellip

52

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382

TRACES OF NONDEFAULT CLASS LATCH

983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152

991270

9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000

991270 E983158983141983150983156 10046 983156983154983137983139983141

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482

PERFORMANCE OF LATCH CLASSES

Row cache objects latch contention testcase on ODA (24 SMT)54

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582

IS THE _SPIN_COUNT REALLY STATIC

bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch

waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used

ldquocache buffers chainrdquo latch contention

bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass

(_LATCH_CLASS_X) By default ndash 20000

bull Shared latch spin can be tuned by _SPIN_COUNT value By

default ndash 2000 However this is not trivial

55

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682

SHARED LATCH BEHAVE LIKE ENQUEUE

bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80

bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests

bull This blocking achieved by special 0x40000000 bit in the latch

value This bit is a flag that some incompatible latch acquisitionis in progress

bull Queued processes will be woken up one by one on latch release

bull X mode latch gets effectively serialize the shared latch

bull See my blog for description of experiments

56

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782

TRACE OF SHARED LATCH WAIT

Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the

latch wait list and then spins again

983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141

983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156

983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983147983155983148983159983148983149983151983140()

983147983155983148983155983147983143983155(098316050009BAC )

983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140

983155983141983149983151983152(27)

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882

SHARED LATCH ACQUISITION

bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned

bull X mode shared latch get spins by default up to 4000 cycles

bull S mode does not spin at all (or spins in unknown way)

S mode get X mode get

Held in S mode Compatible 2_SPIN_COUNT

Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT

58

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982

LATCH RELEASE

bull Free the latch ndash kslfre(laddr)

bull Oracle process releases the latch nonatomically

bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process

bull This requires less bus invalidation and ensures propagation of latch

release to other local cachesbull Not fair policy Spinners on the local CPU board have the

preference

bull Then process posts the first process in the list of waiters

59

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082

LATCH CONTENTION

RDTEX | Latch internals | RUOUG Seminar60

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182

LATCH CONTENTION

bull Occurs when a latch is required by several processes at thesame time

bull Mostly due to bugs or abnormal application behavior

bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp

bull To diagnose the latch contention we need to investigate thelatch statistics [8]

61

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282

DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS

Latch requests arrival rate

Immediate gets efficiency

Latch sleeps ratio

Latch wait time per second

Latch spin efficiencymisses

getsspin∆

∆= _σ

misses

sleepsK

∆=

time

gets

∆=λ

gets

misses

∆= ρ

time

timewait W

∆=

_

Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics

62

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382

SAMPLED AVERAGES

By sampling of x$ksupr we can estimate

ndash Length of latch wait queue

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS

ndash Number of spinning processes in 10g

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS

Sampling of v$lathholder estimates latch utilization

Using DTrace we can measure

ndash Latch acquisition and holding times

ndash Actual spin count

w L

s N

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482

DERIVED LATCH STATISTICS

Latch utilization (PASTA)

Average holding time

Length of latch wait list

Recurrent sleeps ratio

Latch acquisition time

time

timeholdinglatchU

∆=asymp

__ ρ

Re_100

___

questsGet

TimeSnap MissGet Pct S

lowast==

λ

W L =

κ

κ σ 1minus+

)(1

W N T saq += minusλ

64

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 53: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5382

TRACES OF NONDEFAULT CLASS LATCH

983147983155983148983143983141983155(098316050006434 ) 983085 983159983137983145983156 983143983141983156 983151983142 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983089

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983161983145983141983148983140983080983081 983085 983129983145983141983148983140 983090

983155983155983147983143983155983148983155983152983145983150983080983088983160983093983088983088983088983094983092983091983092983081 983086983086983086 983154983141983152983141983137983156983141983140 983089983088983088 983156983145983149983141983155983086

983152983151983148983148983155983161983155983080983086983086983086983084983156983145983149983141983151983157983156983101983089983088 983149983155983084983086983086983086983081 983085 983123983148983141983141983152 983089983086 983109983150983140 983151983142 983155983152983145983150 983137983150983140 983161983145983141983148983140 983148983151983151983152

991270

9831359831489831379831569831399831449831359831399831489831379831559831559831356=100 2 3 10000 20000 30000 40000 50000 60000 70000 80000

991270 E983158983141983150983156 10046 983156983154983137983139983141

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 10886 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=1

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 19922 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=2 983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 30482 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=3

983127AI983124 0 983150983137983149=983148983137983156983139983144 983142983154983141983141 983141983148983137= 39937 983137983140983140983154983141983155983155=991270 983150983157983149983138983141983154=46 983156983154983145983141983155=4

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313504983155983153983148 46 53

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482

PERFORMANCE OF LATCH CLASSES

Row cache objects latch contention testcase on ODA (24 SMT)54

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582

IS THE _SPIN_COUNT REALLY STATIC

bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch

waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used

ldquocache buffers chainrdquo latch contention

bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass

(_LATCH_CLASS_X) By default ndash 20000

bull Shared latch spin can be tuned by _SPIN_COUNT value By

default ndash 2000 However this is not trivial

55

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682

SHARED LATCH BEHAVE LIKE ENQUEUE

bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80

bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests

bull This blocking achieved by special 0x40000000 bit in the latch

value This bit is a flag that some incompatible latch acquisitionis in progress

bull Queued processes will be woken up one by one on latch release

bull X mode latch gets effectively serialize the shared latch

bull See my blog for description of experiments

56

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782

TRACE OF SHARED LATCH WAIT

Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the

latch wait list and then spins again

983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141

983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156

983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983147983155983148983159983148983149983151983140()

983147983155983148983155983147983143983155(098316050009BAC )

983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140

983155983141983149983151983152(27)

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882

SHARED LATCH ACQUISITION

bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned

bull X mode shared latch get spins by default up to 4000 cycles

bull S mode does not spin at all (or spins in unknown way)

S mode get X mode get

Held in S mode Compatible 2_SPIN_COUNT

Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT

58

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982

LATCH RELEASE

bull Free the latch ndash kslfre(laddr)

bull Oracle process releases the latch nonatomically

bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process

bull This requires less bus invalidation and ensures propagation of latch

release to other local cachesbull Not fair policy Spinners on the local CPU board have the

preference

bull Then process posts the first process in the list of waiters

59

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082

LATCH CONTENTION

RDTEX | Latch internals | RUOUG Seminar60

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182

LATCH CONTENTION

bull Occurs when a latch is required by several processes at thesame time

bull Mostly due to bugs or abnormal application behavior

bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp

bull To diagnose the latch contention we need to investigate thelatch statistics [8]

61

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282

DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS

Latch requests arrival rate

Immediate gets efficiency

Latch sleeps ratio

Latch wait time per second

Latch spin efficiencymisses

getsspin∆

∆= _σ

misses

sleepsK

∆=

time

gets

∆=λ

gets

misses

∆= ρ

time

timewait W

∆=

_

Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics

62

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382

SAMPLED AVERAGES

By sampling of x$ksupr we can estimate

ndash Length of latch wait queue

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS

ndash Number of spinning processes in 10g

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS

Sampling of v$lathholder estimates latch utilization

Using DTrace we can measure

ndash Latch acquisition and holding times

ndash Actual spin count

w L

s N

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482

DERIVED LATCH STATISTICS

Latch utilization (PASTA)

Average holding time

Length of latch wait list

Recurrent sleeps ratio

Latch acquisition time

time

timeholdinglatchU

∆=asymp

__ ρ

Re_100

___

questsGet

TimeSnap MissGet Pct S

lowast==

λ

W L =

κ

κ σ 1minus+

)(1

W N T saq += minusλ

64

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 54: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5482

PERFORMANCE OF LATCH CLASSES

Row cache objects latch contention testcase on ODA (24 SMT)54

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582

IS THE _SPIN_COUNT REALLY STATIC

bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch

waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used

ldquocache buffers chainrdquo latch contention

bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass

(_LATCH_CLASS_X) By default ndash 20000

bull Shared latch spin can be tuned by _SPIN_COUNT value By

default ndash 2000 However this is not trivial

55

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682

SHARED LATCH BEHAVE LIKE ENQUEUE

bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80

bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests

bull This blocking achieved by special 0x40000000 bit in the latch

value This bit is a flag that some incompatible latch acquisitionis in progress

bull Queued processes will be woken up one by one on latch release

bull X mode latch gets effectively serialize the shared latch

bull See my blog for description of experiments

56

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782

TRACE OF SHARED LATCH WAIT

Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the

latch wait list and then spins again

983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141

983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156

983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983147983155983148983159983148983149983151983140()

983147983155983148983155983147983143983155(098316050009BAC )

983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140

983155983141983149983151983152(27)

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882

SHARED LATCH ACQUISITION

bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned

bull X mode shared latch get spins by default up to 4000 cycles

bull S mode does not spin at all (or spins in unknown way)

S mode get X mode get

Held in S mode Compatible 2_SPIN_COUNT

Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT

58

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982

LATCH RELEASE

bull Free the latch ndash kslfre(laddr)

bull Oracle process releases the latch nonatomically

bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process

bull This requires less bus invalidation and ensures propagation of latch

release to other local cachesbull Not fair policy Spinners on the local CPU board have the

preference

bull Then process posts the first process in the list of waiters

59

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082

LATCH CONTENTION

RDTEX | Latch internals | RUOUG Seminar60

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182

LATCH CONTENTION

bull Occurs when a latch is required by several processes at thesame time

bull Mostly due to bugs or abnormal application behavior

bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp

bull To diagnose the latch contention we need to investigate thelatch statistics [8]

61

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282

DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS

Latch requests arrival rate

Immediate gets efficiency

Latch sleeps ratio

Latch wait time per second

Latch spin efficiencymisses

getsspin∆

∆= _σ

misses

sleepsK

∆=

time

gets

∆=λ

gets

misses

∆= ρ

time

timewait W

∆=

_

Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics

62

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382

SAMPLED AVERAGES

By sampling of x$ksupr we can estimate

ndash Length of latch wait queue

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS

ndash Number of spinning processes in 10g

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS

Sampling of v$lathholder estimates latch utilization

Using DTrace we can measure

ndash Latch acquisition and holding times

ndash Actual spin count

w L

s N

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482

DERIVED LATCH STATISTICS

Latch utilization (PASTA)

Average holding time

Length of latch wait list

Recurrent sleeps ratio

Latch acquisition time

time

timeholdinglatchU

∆=asymp

__ ρ

Re_100

___

questsGet

TimeSnap MissGet Pct S

lowast==

λ

W L =

κ

κ σ 1minus+

)(1

W N T saq += minusλ

64

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 55: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5582

IS THE _SPIN_COUNT REALLY STATIC

bull Well-known Guy Harrison experiments [5] with Oracle 11gdemonstrated that dynamic _SPIN_COUNT tuning influenced latch

waits CPU and throughputbull The experiments with dynamic change of _SPIN_COUNT used

ldquocache buffers chainrdquo latch contention

bull This latch became shared in Oracle 92bull Exclusive latch uses static SPIN value from x$ksllclass

(_LATCH_CLASS_X) By default ndash 20000

bull Shared latch spin can be tuned by _SPIN_COUNT value By

default ndash 2000 However this is not trivial

55

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682

SHARED LATCH BEHAVE LIKE ENQUEUE

bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80

bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests

bull This blocking achieved by special 0x40000000 bit in the latch

value This bit is a flag that some incompatible latch acquisitionis in progress

bull Queued processes will be woken up one by one on latch release

bull X mode latch gets effectively serialize the shared latch

bull See my blog for description of experiments

56

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782

TRACE OF SHARED LATCH WAIT

Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the

latch wait list and then spins again

983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141

983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156

983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983147983155983148983159983148983149983151983140()

983147983155983148983155983147983143983155(098316050009BAC )

983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140

983155983141983149983151983152(27)

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882

SHARED LATCH ACQUISITION

bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned

bull X mode shared latch get spins by default up to 4000 cycles

bull S mode does not spin at all (or spins in unknown way)

S mode get X mode get

Held in S mode Compatible 2_SPIN_COUNT

Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT

58

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982

LATCH RELEASE

bull Free the latch ndash kslfre(laddr)

bull Oracle process releases the latch nonatomically

bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process

bull This requires less bus invalidation and ensures propagation of latch

release to other local cachesbull Not fair policy Spinners on the local CPU board have the

preference

bull Then process posts the first process in the list of waiters

59

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082

LATCH CONTENTION

RDTEX | Latch internals | RUOUG Seminar60

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182

LATCH CONTENTION

bull Occurs when a latch is required by several processes at thesame time

bull Mostly due to bugs or abnormal application behavior

bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp

bull To diagnose the latch contention we need to investigate thelatch statistics [8]

61

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282

DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS

Latch requests arrival rate

Immediate gets efficiency

Latch sleeps ratio

Latch wait time per second

Latch spin efficiencymisses

getsspin∆

∆= _σ

misses

sleepsK

∆=

time

gets

∆=λ

gets

misses

∆= ρ

time

timewait W

∆=

_

Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics

62

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382

SAMPLED AVERAGES

By sampling of x$ksupr we can estimate

ndash Length of latch wait queue

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS

ndash Number of spinning processes in 10g

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS

Sampling of v$lathholder estimates latch utilization

Using DTrace we can measure

ndash Latch acquisition and holding times

ndash Actual spin count

w L

s N

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482

DERIVED LATCH STATISTICS

Latch utilization (PASTA)

Average holding time

Length of latch wait list

Recurrent sleeps ratio

Latch acquisition time

time

timeholdinglatchU

∆=asymp

__ ρ

Re_100

___

questsGet

TimeSnap MissGet Pct S

lowast==

λ

W L =

κ

κ σ 1minus+

)(1

W N T saq += minusλ

64

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 56: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5682

SHARED LATCH BEHAVE LIKE ENQUEUE

bull Oracle realization of ldquoRead-Writerdquo spinlocks Appeared in 80

bull S and X shared latch modes are incompatiblebull If latch is held in S mode X mode waiter blocks all further requests

bull This blocking achieved by special 0x40000000 bit in the latch

value This bit is a flag that some incompatible latch acquisitionis in progress

bull Queued processes will be woken up one by one on latch release

bull X mode latch gets effectively serialize the shared latch

bull See my blog for description of experiments

56

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782

TRACE OF SHARED LATCH WAIT

Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the

latch wait list and then spins again

983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141

983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156

983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983147983155983148983159983148983149983151983140()

983147983155983148983155983147983143983155(098316050009BAC )

983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140

983155983141983149983151983152(27)

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882

SHARED LATCH ACQUISITION

bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned

bull X mode shared latch get spins by default up to 4000 cycles

bull S mode does not spin at all (or spins in unknown way)

S mode get X mode get

Held in S mode Compatible 2_SPIN_COUNT

Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT

58

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982

LATCH RELEASE

bull Free the latch ndash kslfre(laddr)

bull Oracle process releases the latch nonatomically

bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process

bull This requires less bus invalidation and ensures propagation of latch

release to other local cachesbull Not fair policy Spinners on the local CPU board have the

preference

bull Then process posts the first process in the list of waiters

59

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082

LATCH CONTENTION

RDTEX | Latch internals | RUOUG Seminar60

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182

LATCH CONTENTION

bull Occurs when a latch is required by several processes at thesame time

bull Mostly due to bugs or abnormal application behavior

bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp

bull To diagnose the latch contention we need to investigate thelatch statistics [8]

61

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282

DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS

Latch requests arrival rate

Immediate gets efficiency

Latch sleeps ratio

Latch wait time per second

Latch spin efficiencymisses

getsspin∆

∆= _σ

misses

sleepsK

∆=

time

gets

∆=λ

gets

misses

∆= ρ

time

timewait W

∆=

_

Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics

62

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382

SAMPLED AVERAGES

By sampling of x$ksupr we can estimate

ndash Length of latch wait queue

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS

ndash Number of spinning processes in 10g

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS

Sampling of v$lathholder estimates latch utilization

Using DTrace we can measure

ndash Latch acquisition and holding times

ndash Actual spin count

w L

s N

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482

DERIVED LATCH STATISTICS

Latch utilization (PASTA)

Average holding time

Length of latch wait list

Recurrent sleeps ratio

Latch acquisition time

time

timeholdinglatchU

∆=asymp

__ ρ

Re_100

___

questsGet

TimeSnap MissGet Pct S

lowast==

λ

W L =

κ

κ σ 1minus+

)(1

W N T saq += minusλ

64

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 57: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5782

TRACE OF SHARED LATCH WAIT

Oracle process spins for shared latch twice If the first spin upto _SPIN_COUNT was unsuccessful the process puts itself into the

latch wait list and then spins again

983147983155983148983143983141983156983155983148(098316050009BAC12316) 983085 K983123L GE983124 983123983144983137983154983141983140 L983137983156983139983144 983145983150 983128 983149983151983140983141

983147983155983148983143983141983155983155(098316050009BAC ) 983085 983155983144983137983154983141983140 983148983137983156983139983144 983159983137983145983156 983143983141983156

983147983155983148983155983147983143983155(098316050009BAC )983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983147983155983148983159983148983149983151983140()

983147983155983148983155983147983143983155(098316050009BAC )

983155983155983147983143983155983148983155983152983145983150(098316050009BAC) 983154983141983152983141983137983156983141983140 2000 983156983145983149983141983155

983155983147983143983152983159983159983137983145983156() 983085 983155983148983141983141983152 983157983150983156983145983148 983152983151983155983156983141983140

983155983141983149983151983152(27)

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313505983155983153983148 102 57

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882

SHARED LATCH ACQUISITION

bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned

bull X mode shared latch get spins by default up to 4000 cycles

bull S mode does not spin at all (or spins in unknown way)

S mode get X mode get

Held in S mode Compatible 2_SPIN_COUNT

Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT

58

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982

LATCH RELEASE

bull Free the latch ndash kslfre(laddr)

bull Oracle process releases the latch nonatomically

bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process

bull This requires less bus invalidation and ensures propagation of latch

release to other local cachesbull Not fair policy Spinners on the local CPU board have the

preference

bull Then process posts the first process in the list of waiters

59

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082

LATCH CONTENTION

RDTEX | Latch internals | RUOUG Seminar60

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182

LATCH CONTENTION

bull Occurs when a latch is required by several processes at thesame time

bull Mostly due to bugs or abnormal application behavior

bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp

bull To diagnose the latch contention we need to investigate thelatch statistics [8]

61

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282

DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS

Latch requests arrival rate

Immediate gets efficiency

Latch sleeps ratio

Latch wait time per second

Latch spin efficiencymisses

getsspin∆

∆= _σ

misses

sleepsK

∆=

time

gets

∆=λ

gets

misses

∆= ρ

time

timewait W

∆=

_

Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics

62

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382

SAMPLED AVERAGES

By sampling of x$ksupr we can estimate

ndash Length of latch wait queue

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS

ndash Number of spinning processes in 10g

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS

Sampling of v$lathholder estimates latch utilization

Using DTrace we can measure

ndash Latch acquisition and holding times

ndash Actual spin count

w L

s N

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482

DERIVED LATCH STATISTICS

Latch utilization (PASTA)

Average holding time

Length of latch wait list

Recurrent sleeps ratio

Latch acquisition time

time

timeholdinglatchU

∆=asymp

__ ρ

Re_100

___

questsGet

TimeSnap MissGet Pct S

lowast==

λ

W L =

κ

κ σ 1minus+

)(1

W N T saq += minusλ

64

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 58: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5882

SHARED LATCH ACQUISITION

bull Shared latch spin in Oracle 92-11g is governed by _SPIN_COUNTvalue and can be dynamically tuned

bull X mode shared latch get spins by default up to 4000 cycles

bull S mode does not spin at all (or spins in unknown way)

S mode get X mode get

Held in S mode Compatible 2_SPIN_COUNT

Held in X mode 0 2_SPIN_COUNTBlocking mode 0 2_SPIN_COUNT

58

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982

LATCH RELEASE

bull Free the latch ndash kslfre(laddr)

bull Oracle process releases the latch nonatomically

bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process

bull This requires less bus invalidation and ensures propagation of latch

release to other local cachesbull Not fair policy Spinners on the local CPU board have the

preference

bull Then process posts the first process in the list of waiters

59

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082

LATCH CONTENTION

RDTEX | Latch internals | RUOUG Seminar60

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182

LATCH CONTENTION

bull Occurs when a latch is required by several processes at thesame time

bull Mostly due to bugs or abnormal application behavior

bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp

bull To diagnose the latch contention we need to investigate thelatch statistics [8]

61

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282

DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS

Latch requests arrival rate

Immediate gets efficiency

Latch sleeps ratio

Latch wait time per second

Latch spin efficiencymisses

getsspin∆

∆= _σ

misses

sleepsK

∆=

time

gets

∆=λ

gets

misses

∆= ρ

time

timewait W

∆=

_

Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics

62

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382

SAMPLED AVERAGES

By sampling of x$ksupr we can estimate

ndash Length of latch wait queue

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS

ndash Number of spinning processes in 10g

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS

Sampling of v$lathholder estimates latch utilization

Using DTrace we can measure

ndash Latch acquisition and holding times

ndash Actual spin count

w L

s N

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482

DERIVED LATCH STATISTICS

Latch utilization (PASTA)

Average holding time

Length of latch wait list

Recurrent sleeps ratio

Latch acquisition time

time

timeholdinglatchU

∆=asymp

__ ρ

Re_100

___

questsGet

TimeSnap MissGet Pct S

lowast==

λ

W L =

κ

κ σ 1minus+

)(1

W N T saq += minusλ

64

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 59: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 5982

LATCH RELEASE

bull Free the latch ndash kslfre(laddr)

bull Oracle process releases the latch nonatomically

bull Then it sets up memory_barrier ndash perform atomic operation onaddress individual to each process

bull This requires less bus invalidation and ensures propagation of latch

release to other local cachesbull Not fair policy Spinners on the local CPU board have the

preference

bull Then process posts the first process in the list of waiters

59

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082

LATCH CONTENTION

RDTEX | Latch internals | RUOUG Seminar60

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182

LATCH CONTENTION

bull Occurs when a latch is required by several processes at thesame time

bull Mostly due to bugs or abnormal application behavior

bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp

bull To diagnose the latch contention we need to investigate thelatch statistics [8]

61

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282

DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS

Latch requests arrival rate

Immediate gets efficiency

Latch sleeps ratio

Latch wait time per second

Latch spin efficiencymisses

getsspin∆

∆= _σ

misses

sleepsK

∆=

time

gets

∆=λ

gets

misses

∆= ρ

time

timewait W

∆=

_

Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics

62

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382

SAMPLED AVERAGES

By sampling of x$ksupr we can estimate

ndash Length of latch wait queue

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS

ndash Number of spinning processes in 10g

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS

Sampling of v$lathholder estimates latch utilization

Using DTrace we can measure

ndash Latch acquisition and holding times

ndash Actual spin count

w L

s N

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482

DERIVED LATCH STATISTICS

Latch utilization (PASTA)

Average holding time

Length of latch wait list

Recurrent sleeps ratio

Latch acquisition time

time

timeholdinglatchU

∆=asymp

__ ρ

Re_100

___

questsGet

TimeSnap MissGet Pct S

lowast==

λ

W L =

κ

κ σ 1minus+

)(1

W N T saq += minusλ

64

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 60: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6082

LATCH CONTENTION

RDTEX | Latch internals | RUOUG Seminar60

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182

LATCH CONTENTION

bull Occurs when a latch is required by several processes at thesame time

bull Mostly due to bugs or abnormal application behavior

bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp

bull To diagnose the latch contention we need to investigate thelatch statistics [8]

61

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282

DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS

Latch requests arrival rate

Immediate gets efficiency

Latch sleeps ratio

Latch wait time per second

Latch spin efficiencymisses

getsspin∆

∆= _σ

misses

sleepsK

∆=

time

gets

∆=λ

gets

misses

∆= ρ

time

timewait W

∆=

_

Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics

62

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382

SAMPLED AVERAGES

By sampling of x$ksupr we can estimate

ndash Length of latch wait queue

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS

ndash Number of spinning processes in 10g

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS

Sampling of v$lathholder estimates latch utilization

Using DTrace we can measure

ndash Latch acquisition and holding times

ndash Actual spin count

w L

s N

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482

DERIVED LATCH STATISTICS

Latch utilization (PASTA)

Average holding time

Length of latch wait list

Recurrent sleeps ratio

Latch acquisition time

time

timeholdinglatchU

∆=asymp

__ ρ

Re_100

___

questsGet

TimeSnap MissGet Pct S

lowast==

λ

W L =

κ

κ σ 1minus+

)(1

W N T saq += minusλ

64

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 61: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6182

LATCH CONTENTION

bull Occurs when a latch is required by several processes at thesame time

bull Mostly due to bugs or abnormal application behavior

bull As utilization increases latch waits and spins also increasePerformance degradation can be extremely sharp

bull To diagnose the latch contention we need to investigate thelatch statistics [8]

61

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282

DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS

Latch requests arrival rate

Immediate gets efficiency

Latch sleeps ratio

Latch wait time per second

Latch spin efficiencymisses

getsspin∆

∆= _σ

misses

sleepsK

∆=

time

gets

∆=λ

gets

misses

∆= ρ

time

timewait W

∆=

_

Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics

62

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382

SAMPLED AVERAGES

By sampling of x$ksupr we can estimate

ndash Length of latch wait queue

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS

ndash Number of spinning processes in 10g

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS

Sampling of v$lathholder estimates latch utilization

Using DTrace we can measure

ndash Latch acquisition and holding times

ndash Actual spin count

w L

s N

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482

DERIVED LATCH STATISTICS

Latch utilization (PASTA)

Average holding time

Length of latch wait list

Recurrent sleeps ratio

Latch acquisition time

time

timeholdinglatchU

∆=asymp

__ ρ

Re_100

___

questsGet

TimeSnap MissGet Pct S

lowast==

λ

W L =

κ

κ σ 1minus+

)(1

W N T saq += minusλ

64

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 62: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6282

DIFFERENTIAL (POINT IN TIME) LATCH STATISTICS

Latch requests arrival rate

Immediate gets efficiency

Latch sleeps ratio

Latch wait time per second

Latch spin efficiencymisses

getsspin∆

∆= _σ

misses

sleepsK

∆=

time

gets

∆=λ

gets

misses

∆= ρ

time

timewait W

∆=

_

Should be calculated for each child latch separatelyV$LATCH averaging distorts statistics

62

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382

SAMPLED AVERAGES

By sampling of x$ksupr we can estimate

ndash Length of latch wait queue

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS

ndash Number of spinning processes in 10g

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS

Sampling of v$lathholder estimates latch utilization

Using DTrace we can measure

ndash Latch acquisition and holding times

ndash Actual spin count

w L

s N

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482

DERIVED LATCH STATISTICS

Latch utilization (PASTA)

Average holding time

Length of latch wait list

Recurrent sleeps ratio

Latch acquisition time

time

timeholdinglatchU

∆=asymp

__ ρ

Re_100

___

questsGet

TimeSnap MissGet Pct S

lowast==

λ

W L =

κ

κ σ 1minus+

)(1

W N T saq += minusλ

64

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 63: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6382

SAMPLED AVERAGES

By sampling of x$ksupr we can estimate

ndash Length of latch wait queue

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLAWAT=LADDRESS

ndash Number of spinning processes in 10g

SELECT COUNT(1) FROM X$KSUPR WHERE KSLLALAQ=LADDRESS

Sampling of v$lathholder estimates latch utilization

Using DTrace we can measure

ndash Latch acquisition and holding times

ndash Actual spin count

w L

s N

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482

DERIVED LATCH STATISTICS

Latch utilization (PASTA)

Average holding time

Length of latch wait list

Recurrent sleeps ratio

Latch acquisition time

time

timeholdinglatchU

∆=asymp

__ ρ

Re_100

___

questsGet

TimeSnap MissGet Pct S

lowast==

λ

W L =

κ

κ σ 1minus+

)(1

W N T saq += minusλ

64

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 64: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6482

DERIVED LATCH STATISTICS

Latch utilization (PASTA)

Average holding time

Length of latch wait list

Recurrent sleeps ratio

Latch acquisition time

time

timeholdinglatchU

∆=asymp

__ ρ

Re_100

___

questsGet

TimeSnap MissGet Pct S

lowast==

λ

W L =

κ

κ σ 1minus+

)(1

W N T saq += minusλ

64

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 65: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6582

LATCH STATISTICS EXAMPLEL983137983156983139983144 983155983156983137983156983145983155983156983145983139983155 983142983151983154

0983160380007358 983155983141983155983155983145983151983150 983137983148983148983151983139983137983156983145983151983150

983122983141983153983157983141983155983156983155 983154983137983156983141 983148983137983149983138983140983137= 1350 H983162

M983145983155983155 983143983141983156 983154983144983151= 022

983123983137983149983152983148983141983140 983125983156983145983148983145983162983137983156983145983151983150 983125= 013

983123983148983152983155 M983145983155983155 983147983137983152983152983137= 28

983127983137983145983156983135983156983145983149983141983155983141983139 983127= 021

983123983137983149983152983148983141983140 983153983157983141983157983141 983148983141983150983143983156983144 983116983159= 017983123983152983145983150983135983143983141983156983155983149983145983155983155 983155983145983143983149983137= 72

983123983137983149983152983148983141983140 983155983152983145983150983150983145983150983143 983152983154983151983139983155983118983155= 013

983123983141983139983151983150983140983137983154983161 983155983148983141983141983152983155 983154983137983156983145983151 = 002

A983158983143 983144983151983148983140983145983150983143 983156983145983149983141= 163 983157983155983141983139983155983148983141983141983152983145983150983143 983156983145983149983141 = 159 983157983155983141983139

983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 = 258 983157983155983141983139

L983137983156983139983144 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141 983140983145983155983156983154983145983138983157983156983145983151983150

983149983141983137983155983157983154983141983140 983138983161 D983124983154983137983139983141------------------------------------ DistributionDistributionDistributionDistribution --------------------------------

2048 |2048 |2048 |2048 |4096 |4096 |4096 |4096 |

8192 |8192 |8192 |8192 |

16384 |16384 |16384 |16384 |

32768 |32768 |32768 |32768 |

65536 |65536 |65536 |65536 |

nsnsnsns

A983158983141983154983137983143983141 983137983139983153983157983145983155983145983156983145983151983150 983156983145983149983141=21 983157983155983141983139

983155983153983148983152983148983157983155 983150983151983148983151983143 98314098314198314998315198313506983155983153983148 65

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 66: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6682

LATCH CONTENTION DIAGNOSTICS IN 92-11G

bull Should be suspected if latch wait events are observed in ldquoTop 5 TimedEventsrdquo AWR section

bull Look for the latch with highest W

bull Symptoms of contention for the latch

ndash W gt 1 secsec

ndash Utilization gt 10

ndash Acquisition (or sleeping) time sufficiently greater then holding time

bull Latchprofxsql script invented by Tanel Poder [4] greatly simplifiesdiagnostics

bull The script and v$latch_misses reveal ldquowhererdquo the contention arise

ρ

66

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 67: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6782

LATCH LEVELS AND LATCH CONTENTION

bull Process holding latch can only acquire latch with higher level

bull Contention for a high-level latch frequently exacerbates contentionfor lower-level latches

bull ldquoLibrary cacherdquo latch contention without many cursor versionslikely is a consequence of moderate shared pool latch contention

bull Shared pool resizes clearances and ORA-4031 dumps can induceldquolibrary cacherdquo latch storms

bull Use latch_treesql script to confirm this scenario

67

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 68: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6882

BEWARE OF CERTAIN V$X$

bull Frequent scans of some x$ tables induce latch contention

ndash x$ktcxb (v$transaction) - ldquoTransaction allocation rdquo latch ndash x$ktadm (v$lock dba_jobs_running) ndash DML lock allocation

ndash x$kqlfxpl (v$sql_plan) ndash library cache latch in 10g

ndash x$ksmsp ndash shared pool latch

ndash x$kslltr (v$latch) ndash all parent latches in 11g

ndash hellip

68

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 69: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 6982

TREATING THE LATCH CONTENTION

bull The right method tune the application and reduce the latchdemand Tune the SQL bind variables schema etchellip

bull Many brilliant books exist on this topic Out of scope for thispresentation

bull Install latest Oracle PSU and OS recommended patches Severallatch related examples

ndash Oracle bug 7627743 ldquoHigher latch waits in 11grdquo Processes in 111always spun upto the maximum spin count value

ndash Linux 269 kernel bug 149933 Missing wakeup in ipcsem Fixed

in Red Hat EL 4 Update 4 ndash HP-UX 1131 Should install scheduler cumulative patch

PHKL_38397 or later

69

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 70: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7082

WHEN WE NEED TO TUNE SPIN COUNT

bull The ldquorightrdquo method to tune the application may be too expensive andrequire complete application rewrite

bull Nowadays the CPU power is cheaper We may already have enough freeCPU resources

bull Oracle does not explicitly forbid _SPIN_COUNT tuning However changeof undocumented parameter should be discussed with Support For

example MOS recommends _SPIN_COUNT=5000 for 10g Streamsbull Processes spin for exclusive latch spin upto 20000 cycles for shared latch

upto 4000 cycles and infinitely for pre 11202 mutex Tuning may findmore optimal values for your application

bull In CPU thrashing decrease of the number of spins can reduce CPUconsumption

70

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 71: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7182

TUNING THE SPIN COUNT EFFICIENTLY

bull Beware of side effects You should have enough free CPU

bull First diagnose the root cause of latch contention Measure the

latch statistics install relevant patches

bull Spin count tuning will only be effective if latch holding time Sis in its normal microseconds range

bull The number of spinning processes should remain less then numberof CPUs Beware CPU thrashing Analyze AWR and latch statisticsbefore and after each change

bull Spin count adjustment is different for exclusive and shared latches

bull Nonstandard latch classes may decrease CPU consumption

bull It is a common myth that CPU time will raise infinitely while weincrease spin count

71

SPIN COUNT ADJUSTMENT

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 72: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7282

SPIN COUNT ADJUSTMENT

Shared latches

bull Dynamically by _SPIN_COUNT parameter

bull Good starting point is the multiple of default 2000 value

bull Setting _SPIN_COUNT parameter in initialization file should beaccompanied by _LATCH_CLASS_0=20000 Otherwise spin forexclusive latches will be greatly affected by next instance restart

Exclusive latches

bull Statically by _LATCH_CLASS_0 parameter Needs the instance restart

bull Good starting point is the multiple of default 20000 value

bull It may be preferable to increase the number of yields for class 0 latchesSpin count tuning probes the tail of latch holding time distribution See my

blog for related mathematics

72

SPIN SCALING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 73: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7382

SPIN SCALING

bull My previous work [8] proposed approximate scaling rules to estimateeffect of SPIN_COUNT tuning depending on

K = sleeps ratio=ldquoAvg SlpsMissrdquo

bull If spin is inefficient then

Doubling the spin count will reduce sleeps ratio by 2 and doubles

the spin CPU consumption

bull If latch holding time distribution has exponential tail you have enoughCPU resources and spin is efficient then

Doubling the spin count will square the ldquosleeps ratiordquo coefficientThis will multiply the spin CPU consumption by

If the spin is already efficient it is worth to increase the spin count

)1( K +

10geK

10ltltK

I983142 983155983148983141983141983152983155 983154983137983156983145983151 983142983151983154 983141983160983139983148983157983155983145983158983141 983148983137983156983139983144 983145983155 10 983156983144983137983150 983145983150983139983154983141983137983155983141 983151983142983155983152983145983150 983139983151983157983150983156 983156983151 40000 983149983137983161 983154983141983155983157983148983156983155 983145983150

10 983156983145983149983141983155 983140983141983139983154983141983137983155983141 983151983142 983148983137983156983139983144 983142983154983141983141 983159983137983145983156 983141983158983141983150983156983155 983137983150983140 983151983150983148983161 10

983145983150983139983154983141983137983155983141 983151983142 C983120983125 983139983151983150983155983157983149983152983156983145983151983150983129983151983157983154 983149983145983148983141983137983143983141 983149983137983161 983158983137983154983161

73

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 74: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7482

CACHE BUFFER CHAINS LATCH CONTENTION TESTCASE

L983137983156983139983144 983159983137983145983156 983137983150983140 C983120983125 983156983145983149983141983155 983158983155 983135983123983120IN983135CO983125N983124

74

LONG LATCH HOLDING TIME CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 75: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7582

LONG LATCH HOLDING TIME CPU THRASHING

bull Latch contention can cause CPU starvation Processes contendingfor a latch also contend for CPU

bull Once CPU starves OS runqueue length raise and loadaverageexceeds the number of CPUs Some OS may shrink the timequantum Latch holders will not receive enough time to release thelatch

bull Due to priority decay latch acquirers may preempt latch holdersThis leads to priority inversion The throughput falls

bull Transition to this metastable state is more likely if workload of

your system approaches ~100 CPUbull Due to preemption latch holding time S will raise to the CPU

scheduling scale [7]

75

CPU THRASHING

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 76: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7682

CPU THRASHING

bull If CPUs are running at 100 utilization for other reasons the latchcontention can be just a symptom of this CPU starvation itself

All wait times will be inflated by wait for CPU However latchholding time should be in normal range

bull CPU thrashing is unlikely on Solaris due to latch preemptioncontrol

bull To prevent the priority inversion use

ndash FX priority class on Solaris

ndash HPUX_SCHED_NOAGE on HPUX

ndash SCHED_BATCH policy on Linux

76

LATCH SMP SCALABILITY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 77: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7782

LATCH SMP SCALABILITY

bull If latch utilization is in single CPU environment

bull Then in N CPU server latch utilization will be This can be

problematic

ndash If single CPU system held latches only for 1 of time

ndash 48 CPU server with the same per-CPU load will hold latches for 50

ndash 128 CPU Cores server will suffer huge latch (and mutex) contention

bull This is also known as Software lockout It may substantially affect

contemporary multi-core serversbull NUMA should overcome this intrinsic spinlock scalability restrictions

1 ρ

1 ρ N N asymp

77

BIBLIOGRAPHY

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 78: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7882

BIBLIOGRAPHY

1 Steve Adams Oracle8i Internal Services for Waits Latches Locks andMemory 1999

2 Tanel Poder blog httpblogtanelpodercom

3 Jonathan Lewis Oracle Core Essential Internals for DBAs andDevelopers 2011

4 Edsger Dijkstra Solution of a Problem in Concurrent ProgrammingControl CACM 1965

5 M Herlihy N Shavit The Art of Multiprocessor Programming 20086 B Sinharoy et al Improving Software MP Efficiency for Shared

Memory Systems Proc of 29th Hawaii Conference on System Sciences1996

7 Guy Harrison blog Using _spin_count to reduce latch contention in 11g

httpguyharrisonsquarespacecom 20088 R Johnson et al A new look at the roles of spinning and blocking

Proc of 5th Workshop on Data Management on New Hardware 2009

9 A Nikolaev Exploring Oracle RDBMS latches using Solaris DTraceProc of MEDIAS 2011 Conf httparxivorgabs11110594v1 2011

78

QA

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 79: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 7982

QA

bull Questions

bull Comments

79

ACKNOWLEDGEMENTS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 80: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8082

ACKNOWLEDGEMENTS

bull Thanks to RDTEX Technical Support Centre Director SPMisiura for years of encouragement and support of my

investigationsbull Thanks to my colleagues for discussions

bull Thanks to all our customers for participating in latch

troubleshooting

80

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 81: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8182

A983150983140983154983141983161 N983145983147983151983148983137983141983158

983144983156983156983152983137983150983140983154983141983161983150983145983147983151983148983137983141983158983159983151983154983140983152983154983141983155983155983139983151983149

A983150983140983154983141983161N983145983147983151983148983137983141983158983154983140983156983141983160983154983157

983122D983124E983128 M983151983155983139983151983159 983122983157983155983155983145983137

983159983159983159983154983140983156983141983160983154983157

THANK YOU

RDTEX | Latch internals | RUOUG

Seminar

IN MEMORY OF MY FRIEND FOR 30 YEARS

RDTEX | Latch internals | RUOUG Seminar

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82

Page 82: Andrey Nikolaev Latch Internals 2012 for Ruoug

7232019 Andrey Nikolaev Latch Internals 2012 for Ruoug

httpslidepdfcomreaderfullandrey-nikolaev-latch-internals-2012-for-ruoug 8282

A983150983140983154983141983161 K983154983145983157983155983144983145983150C983144983137983145983154983149983137983150 983151983142 983122983157983155983155983145983137983150 O983154983137983139983148983141 983125983155983141983154 G983154983151983157983152 (983122983157O983125G)

983108983145983141983140 983151983142 983144983141983137983154983156 983137983156983156983137983139983147 983088983090983086983088983096983086983090983088983089983089

82