Embed Size (px)
Citation preview
April 21 – 23. Neuss, Germany
w w w . s q l p a s s . o r g / e u 2 0 1 0
Inside SQL Server Wait Types
Bob Ward, Principal Escalation Engineer,
Microsoft Corporation
SQL 2005 and SQL 2008
Inside SQL Server Wait Types
Welcome to My World
This is a “500”
level talk
I assume SQL
knowledge
DMVs, Code,
Debugger, and APIs
We will move fast and furious
This means your brain
may hurt
Stuff you can use
Stay for questions as long as you want
=
Inside SQL Server Wait Types
What is a Wait Type?
We created this to help us find bottlenecks
In a galaxy, far, far, away we had locks, I/O and network
But as time has moved on… we went a bit overboard
The name of the type is up to the developer
• I/O, Network, Thread, MemoryResource• Locks, Latches, and “bunch of others”Synchronization• Yield or SleepForced• PreemptionExternal• Background tasksQueue
485 in SQL Server 2008
Not the best of docs
Inside SQL Server Wait Types
How does it work?
Request LCK_M_IS (Shared Intent)
lockA conflict exists
Setup a SOS_WaitInfo with
LCK_M_IS
Call LockOwner::Sleep
Use SOS_EventAuto
class to wait
Understands SQLOS scheduling
Wait() results in SignalObjectAndWait()
We know we need to wait
SOS_EventAuto is a wrapper for Windows
Kernel Event object
Common for a SELECT
Ultimately it always comes down to WaitForSingleObject()
or SignalObjectAndWait()
Inside SQL Server Wait Types
Where do you see Wait Types?
sys.dm_os_wait_stats
sys.dm_exec_requests
sys.dm_os_waiting_tasks
sys.sysprocesses
Extended Events
Management Data Warehouse
Activity Monitor
Performance Monitor Counters
Historical stats
Live state
Legacy
Tracing in 2008
In the tools
Wait Statistics Counter
Inside SQL Server Wait Types
last_wait_type explained
Last wait type used to be stored with session
Now stored with the worker
Only valid while worker bound to task
Last_wait_type = wait_type when task is
waiting
“true last wait type” when task is still
running but not waiting
Pre-emptive waits can confuse this but we will
explain later
Good example is SOS_SCHEDULER_YIELD
Inside SQL Server Wait Types
Dive into Wait Types
Inside SQL Server Wait Types
Common Wait Types
LCK_XX PAGELATCH and PAGEIOLATCH
ASYNC_NETWORK_IO
Locks - sync
BUF latch - sync
Hint: Network or your app
Hint: I/O
delay
Hint: Your app
Make up ~50 of the wait
types
Resource
Hint: System table or
allocation
Inside SQL Server Wait Types
Some Waits may not be bottlenecks
MISCELLANEOUS
Background Task Waits• LAZYWRITER_SLEEP• SQLTRACE_BUFFER_FLUSH• LOGMGR_QUEUE• CHECKPOINT_QUEUE• REQUEST_FOR_DEADLOCK_SEARCH
CLR_AUTO_EVENT
BOL calls these
Queue Waits
Should be called “not waiting”
Normal for SQL CLR
Inside SQL Server Wait Types
Busting the Myth of CXPACKET
Used to synchronize parallel query
workers
Just means you have a parallel query
Do you expect parallel queries?
Do you have high wait times?
wait_resource shows coordination
High wait times mean long running
parallel queries
Look at the Tasks
Which one is not CXPACKET?
Some other wait may be the issue
What Should I Do?
You may not need to do anything
Find queries and tune them
Use the MAXDOP query hint
Modify ‘max degree of parallelism’
SyncCraig Freedman Paper
is a must read
Don’t panic. Some think MAXDOP=1 should be on OLTP systems
Inside SQL Server Wait Types
The Non Buffer Latch
• Not just for BUFs• Thread sync of memory structure
Latch can be generic
• As opposed to PAGELATCH or PAGEIOLATCH
Appears as LATCH_XX
• sys.dm_os_latch_stats• sys.dm_exec_requests.wait_resource
How many are there?
Same modes as BUFs (KP, SH, UP, EX, DT)
Latch class
Sync
Inside SQL Server Wait Types
FGCB_ADD_REMOVE Latch
mydb.mdfAutogrow
INSERT
INSERT
INSERT
INSERT
FGCB
Need space
Need space
SQL Server Engine
Need space
Need space
“I need to grow” LATCH_EX:
FGCB_ADD_REMOVE
LATCH_SH: FGCB_ADD_REMOVE
Sync
Moral of the story: Use
instant file initialization but…it doesn’t
work for the tlog
Inside SQL Server Wait Types
SOS_SCHEDULER_YIELD
A task that does not “naturally” wait must
yield
• When quantum of 4ms is exceeded
• What if we don’t do this right?
Examples
• No I/O needed for pages
• T-SQL variables only or just “expressions”
• Query compile• Small hashes and
sorts
Indicators
• High count/low wait this is actually good
• High count/High wait CPU queries competing
• Low count/High wait someone not yielding or preemptive threads hogging CPU
************************* * BEGIN STACK DUMP:* 10/17/09 15:51:52 spid 0** Non-yielding Scheduler*************************
I/O, Lock, Latch Forced
Inside SQL Server Wait Types
THREADPOOL
Receive TDS packet
Engine creates SQLOS
Task
Find available worker on scheduler
If none, set THREADPOOL
wait type
Next available worker runs
task
TDS Login
Login Timeout
Only seen in stats and tasks
You may need DAC to see it live
PENDING tasks and work_queue_count
in schedulers > 0
Applies to any task
These are pure victims
Look for other waits
Often a long blocking chain
DO NOT assume you need more worker threads
Resource
Request = task + worker
Inside SQL Server Wait Types
What about I/O Waits?
Log Cache
Mylog.ldf
Log WriterCOMMIT TRAN
Log BufferINSERT
Flush Log Buffer
Request Log BufferAll buffers in
use
LOGBUFFER
WRITELOG
File
Copy model
SQLTrace File
Sort I/O
IO_COMPLETION
Backup reading files
Create database files ASYNC_IO_COMPLETION
Mylog.ldf and .mdf
Backup media
VDI AppEngine
Workers
Page I/O
BACKUP WITH SNAPSHOT
DISKIO_SUSPEND
Resource
Sync
Resource
Resource
Sync
Inside SQL Server Wait Types
Queries, Memory, and RESOURCE semaphores
• Limited memory or too many concurrent users• MEMORYCLERK_SQLQUERYEXEC and
MEMORYCLERK_SQLQERESERVATIONS clerks• dm_exec_query_resource_semaphores• dm_exec_query_memory_grants• RESOURCE_SEMAPHORE_SMALL_QUERY waits
RESOURCE_SEMAPHORE(Query Memory)
• Why are you compiling so much?• Factor of limited memory or “memory hungry” compiles• Throttled on a system of levels (gateways) with
thresholds• High Query Memory lowers thresholds• Not often seen on 64bit systems
RESOURCE_SEMAPHORE_QUERY_COMPILE
Resource
sys.dm_os_memory_brokersDBCC MEMORYSTATUS
Hashes and sorts
compiles
Inside SQL Server Wait Types
Pre-Emptive Waits
• Windows API• Xproc
Workers go pre-emptive when calling “external” APIs that may take “some time”
• Status = RUNNING• Wait_type = NULL
What does this look like pre-SQL 2008?
• Status = RUNNING• Wait_type = PREEMPTIVE_XXXX
What does this look like in SQL 2008?
• We can go pre-emptive while another wait type is set (Ex. ASYNC_NETWORK_IO)
Can see along with “normal” wait type
External
************************* * BEGIN STACK DUMP:* 10/17/09 15:51:52 spid 0** Non-yielding Scheduler*************************
May wrap more code than just
the API
Inside SQL Server Wait Types
What are some I might see?
Type Description Scenario
PREEMPTIVE_OS_GETPROCADDRESS Wraps calls to GetProcAddress() and xproc function
Measure of xproc execution time
PREEMPTIVE_OS_WRITEFILEGATHER Wraps calls to WriteFileGather() to zero out a section of a file
Long autogrow for tlog file or database files (if not using instant file init)
PREEMPTIVE_OS_LOOKUPACCOUNTSID Wrapped calls to LookupAccountSid()
Mostly used during login authentication. Long waits could indicate DC issues.
PREEMPTIVE_OLEDBOPS Wrapped around various code fragments that will call OLE-DB methods for linked server queries..
Helps fill in gaps where OLEDB wait not set.
PREEMPTIVE_OS_WAITFORSINGLEOBJECT Wrapped around WaitForSingleObject() calls in network I/O and waiting for recovery to complete
Can sometimes be seen in conjunction with ASYNC_NETWORK_IO
~190 of these
Inside SQL Server Wait Types
Extended Events and Waits
wait_info
“Normal” waits
wait_info_external
Pre-emptive
waits
wait_type
opcode
Timings
Get query, session, or stack dump
On by Default System_Health Session Has These
SQLCAT Waits Stats Per Session Project
Begin and End
Duration (Total and Max are per scheduler)
Inside SQL Server Wait Types
Where is THE LIST?
• and in sys.dm_xe_map_values for SQL Server 2008
In a header file in the source code
The BOL list
KB article on waittypes is only for SQL 2000 and prior
• Post new findings on this blog post• Comment on the blog or send email to [email protected]• Use the blog to update the BOL• Blog may contain scenarios and more details
The planThe Wait Type Repository
Blog
Inside SQL Server Wait Types
Resources
Our CSS Escalation Blog
The Wait Type Repository Blog Post
BOL reference on sys.dm_os_wait_stats
SQLCAT Waits Stats Per Session CodePlex
Craig Freedman blog posts on Parallelism
CLR Wait Types blog post
SQL Server 2005 Waits and Queues Whitepaper
The System_Health XEvent Session Blog
Questions
Inside SQL Server Wait Types
Appendix
Inside SQL Server Wait Types
The General Algorithm
Developer writes code that “runs”
Developer knows they might execute code
that “waits”
Developer “sets” a wait type
Developer calls SQLOS routines to “wait”
Anyone querying the DMVS sees the wait
type and accumulated wait time
Code is signaled to “wake-up”
Code clears wait type and time.
last wait type in worker not cleared
until task is complete
Inside SQL Server Wait Types
What does MDW tell you about I/O waits?
sync reads, sorts, SQLTrace I/O, load CLR
assembly
Backups, Recovery, DBM
Buffer Pool I/O for pages
WRITELOG wait time =Log Flush Wait (perfmon)
LOGBUFFER is just waiting on folks waiting on
WRITELOG
25
Inside SQL Server Wait Types
The Mapping for sysprocesses has changed
sys.dm_xe_map_values is to be used for the mapping
Binary to string mapping changed in SQL 2005
sys.sysprocesses.waittype is a binary value
KB 822101 wrong for 2005 and 2008
Inside SQL Server Wait Types
What about these?
• Fixed time• Hard to figure out scenario
SLEEP_TASK
• Log shipping delayed to secondary
DBMIRROR_DBM_EVENT
• Wrapped around linked server OLE-DB API calls• Wait time will fluctuate since set and cleared for each call• wait_resource is remote server and remote SPID• PREEMPTIVE_XX type can now also show up
OLEDB
• Thread synchronization for memory allocation• High wait times = A likely bug
CMEMTHREAD
Forced
Resource
External
Sync
Hot stored proc in SQL Server 2005
27
Inside SQL Server Wait Types
The are “other” waits
log_reuse_wait “loader lock wait” sessions
Spinlocks Resource Governor
backoffs in sys.dm_os_spinlock_stats
“Why can’t I truncate the
log”
A poorly written DLL
PRECONNECT status
You decide to throttle
Thank you!
Don’t forget to fill out your session evaluation forms
