Sql server 2005 memory internal

"Technology is dominated by two types of people: those who understand what they do not manage, and those who manage what they do not understand.“

Putt's Law and the Successful Technocrat: How to Win in the Information Age

DAT322: SQL Server 2005 Memory InternalGeyzerskiy DmitriyChief Architect Microsoft Technologies

dimag@dbnet.co.il

Session Objectives and Agenda• Windows Memory Management

• NUMA Architecture

• SQL Server Memory Management

• AWE vs. 64-bit

Myths

Windows Memory Overview• Virtual Memory

• 32 bits: 4GB• 64 bits: 17,179,869,184 GB• VM states: Committed, Reserved, Free

• Page size• 4KB on x86, EM64T, AMD64• 8KB on Itanium• Large pages: 4MB on x64, 16MB on Itanium

• Paging• Page Faults• Working Sets• Kernel Memory And User Memory

Retrieving system information using Win32 API

3GB Process Space Option

• /3GB switch in BOOT.INI

• /USERVA (between 2048 and 3072, in 128MB increments)

• .EXE must be linked with LARGEADDRESSAWARE flag

64 bit Address Space

• Map more data into the address space

• The application “speed” is the same on 32 bit and

64 bit• OS needs 2GB of memory to hold pointers to 16GB or

more physical memory

x64 IA64

Paging Dynamics

Working Sets

Standby Standby ListList

ModifiedModifiedListList

FreeFreeListList

ZeroZeroListList

Sizing the Page File• More RAM should mean smaller page file!

• Crash dump settings affect

• Full: size of RAM

• Kernel dump: much smaller

• To size correctly, review what goes there

• Minimum should = commit charge peak

• Maximum could be a multiple of this

Measuring memory performance

Agenda

• Windows Memory Management

• NUMA Architecture

• SQL Server Memory Management

• AWE vs. 64-bit

NUMA

What is SMP

• SMP – Symmetric Multi-Processing• Front-bus point of contention• Difficult to scale beyond 32 CPU

Front-bus

What is NUMANUMA (Non-Uniformed Memory Access)

Local Memory Access

Foreign Memory

Access

4x local Local Memory Access

What is Interleaved-NUMA

• Enable NUMA hardware to behave as SMP• Memory are used by all CPUs

• Each CPU’s cache line access slice of memory from all nodes

• SQL Server 2000 should use interleaved-NUMA

Local Memory Access Foreign Memory Access

What is Soft-NUMA

• Activates custom NUMA configuration on top of any of hardware• Registry settings control final

SoftNUMA configuration• Provides greater performance,

scalability, and manageability on SMP as well as on real NUMA hardware

Soft-NUMA Configuration Example

We have: NUMA system with 2 nodes and 4 CPU per Node

We need:2 CPUs for loading application and the rest of CPUs for queries.

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\90\NodeConfiguration\Node0]"CPUMask"=dword:0000000F[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\90\NodeConfiguration\Node1]"CPUMask"=dword:00000030[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\90\NodeConfiguration\Node2]"CPUMask"=dword:000000C0

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL.1\MSSQLServer\SuperSocketNetLib\Tcp\IPAll]"TcpPort"="1433[0x3],2000[0x4]""TcpDynamicPorts"="""DisplayName"="Any IP Address"

Soft-NUMA Configuration Example

Agenda

• Windows Memory Management

• NUMA Architecture

• SQL Server Memory Management

• AWE vs. 64-bit

Server Architecture

DeadlockMonitor

SQLO

S Hosting API

Scheduling

Memory Manager

BufferPool

LockManager

Synchronization ResourceMonitor

LazyWriter

SQLOS API

I/O

SQLO

S API

External Components (CLR/M

DAC)

SQLOS

SchedulerMonitor

= thread

Parser Optimizer SQL Manager

Database Manager

Query Executor

Query Processor

Transaction Services

Lock Manager

File Manager

Buffer Manager

Utilities: Bulk Load DBCCBackup/Restore

Access Methods Managers for:

Row OperationsIndexesPagesAllocationsVersions

Storage Engine

Protocols

SQL Server Memory Management

Resource Monitor

Generic Memory

Clerk

Cache Memory

Clerk

Buffer Pool Memory

ClerkCLR Memory

Clerk

Memory Node

Low Physical Internal/External Low VAS High Physical Internal/External

CP CacheStore

(@1 numeric(2,1), @2 numeric(3,2)) insert [t] values(@1 , @2)

insert t values (2.1, 3.12) insert t values (3.33, 4.1)

(@1 numeric(3,2), @2 numeric(2,1)) insert [t] values(@1 , @2)

Auto-parameterization

Bucket 0 Bucket 1 Bucket 2

SQL Server 2005 RTM SQL Server 2005 SP2

Monitoring Memory PressureLast notification RM broadcasted

select * from sys.dm_os_ring_bufferswhere ring_buffer_type=‘RING_BUFFER_RESOURCE_MONITOR‘

BP/Single page allocator turns on/off internal memory pressure indicator

select * from sys.dm_os_ring_bufferswhere ring_buffer_type=‘RING_BUFFER_SINGLE_PAGE_ALLOCATOR‘

Single-page allocation vs. multi-page allocation

select * from sys.dm_os_memory_cache_counters

SQL Server Memory DMVs

Agenda

• Windows Memory Management

• NUMA Architecture

• SQL Server Memory Management

• AWE vs. 64-bit

Address Windowing Extensions (AWE)• Access more than 4GB of physical memory.• Is ignored on systems with less than 3GB of physical

memory.• Is never swapped to disk.

Allocate the physical memory (Lock Pages in Memory)1

Create a region in the process address space to serve as a window for mapping views of this

physical memory2

Map a view of the physical memory into the virtual memory window3

SQL Server Process Address Space with AWE

0xFFFFFFFF

0x00000000

0x80000000

0xC0000000

Operating System

SQL Server or OS(/3GB switch)

SQL

Serv

er

MemToLeave areaThread stacks

Oth

er

Lock

s

Que

ry

Wor

kspa

ce

Plan

Cac

he

DB

Pag

e C

ache

(Buf

fer P

ool)

AWE Memory

SQL Server 2005 32-bit AWE Memory• Right OS version

• Windows Server 2003 Standard and up

• /PAE in boot.ini enables 32-bit OS to address more than 4GB memory

• SQL Server Edition• Enterprise Edition • Developer Edition

• sp_configure ‘awe enabled’

Lock Pages In Memory Option

• Entry in the SQLERROR log• 64 bit: Using locked pages for buffer pool• 32 bit: Address Windowing Extensions is enabled

• Discarded in Standard Edition

• The Local System account has the 'lock pages in memory' privilege by default

A significant part of sql server process memory has been paged out. This may result in a performance degradation. Duration: 0 seconds. Working set (KB): 1086400, committed (KB): 2160928, memory utilization: 50%.

SQL Server 2005 64 bit vs. 32 bit

• The only way to get virtual memory > 3GB

• What is different from 32-bit?

• All pointers are 64-bit

• SQL Server commits ‘min server memory’ memory at startup

• Some internal memory-related data-structure constants

larger

• 64-bit alignment of data structures

SQL Server 2005 64 bit vs. 32 bit

• What is the same?

• No on-disk database format changes

• No differences in buffer pool policy / algorithms from 32-bit

• All uses of memory can use additional 64-bit

memory

• DB Page Cache, Query Workspace Memory, Plan Cache,

Locks, External uses, Utilities, …

32-Bit and 64-Bit System Limitations

Resources• Blogs

• Slava Oks’ blog: http://blogs.msdn.com/slavao• SQL Programmability & API Development Team Blog:

http://blogs.msdn.com/sqlprogrammability/

• External Links:• NUMA FAQ: http://lse.sourceforge.net/numa/faq

• Books:• Eldad Eilam: Reversing: Secrets of Reverse

Engineering• Ken Henderson: SQL Server 2005 Practical

Troubleshooting The Database Engine • Kalen Delaney: Inside Microsoft SQL Server 2005 The

Storage Engine

Summary• It pays to understand SQL Server memory management

• A number of performance issues either originate or manifest as memory issues

• Memory-based performance tuning is a very useful technique • Significant internal and external changes in SQL Server 2005

• Consider NUMA for your next large-scale project

• Upgrade your system to 64 bit

© 2007 Microsoft Corporation. All rights reserved.This presentation is for informational purposes only.MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS SUMMARY.

Backup Slides

Memory

Memory Subsystem

Procedure,Http, etc.Caches

Memory Clerk

Node Memory

Subsystem

Memory ObjectFactory

CacheStore

MemoryObjects

SOS’ Memory ManagerSQL Memory Components

Buffer Pool

Database Page Cache

• Most common use of memory - often referred to as “Buffer Pool”

• Stores database pages – index, heaps• Lazy writer thread sweeps across buffer pool to age

pages out of cache• Uses a modified clock algorithm

• Each page has a reference count• Reference count divided by 4 each time clock hand passes• Pages with reference count 0 can be discarded

• A Dirty page needs to be written out first• Favors keeping often-used pages in cache• Higher level index pages naturally favored

• Full scans may do some damage to buffer pool • Clock algorithm limits damage

Monitoring Database Page Cache

• Perfmon: SQL Server:Buffer Manager• Buffer cache hit ratio: SQL 2000 SP3 onwards, this is

“recent” data (last 2K to 3K accesses)• Page life expectancy: low value (< 300 seconds)

indicates “churn” in buffer pool• Physical Disk: Avg Disk sec/read, write transfer

• Beware perfmon averaging • Free list stalls/sec: another indication of memory

pressure • AWE counters – may correlate to kernel time

• Related: Per file I/O statistics obtained via ::fn_virtualfilestats(dbid, fileid)• IoStallMS shows file-level I/O bottleneck

Plan Cache

• Caches plans for various types of batches• Stored procedures, Triggers, Ad-hoc SQL, Auto-

parameterized SQL, Parameterized SQL (sp_executesql or via client APIs)

• Plans are of two types• Compiled plan

• Read-only• One per unique combination of statement, dbid, unique

combination of set options• Executable plan / Execution Context

• Derived from compiled plan – points back to it• One per concurrent execution• Contains execution specific data – e.g. parameter/row values• Not all executable plans cacheable – e.g. hash, parallel plans

• No pre-defined upper limit for size of plan cache• Depends on buffer pool to manage space

Monitoring Plan Cache

• master.dbo.syscacheobjects• Lists all items in plan cache• Can aggregate this data to get use counts• Very useful indicator of nature of application

• dbcc proccache• High level summary data on plan cache

• dbcc cachestats• Summary by cache object type

• Perfmon counters under Cache Manager• Counts of cache pages, objects• However, hit ratios are from instance startup

• Perfmon counters under SQL Statistics• Can monitor compiles, recompiles, etc.

• Profiler Events • SP:CacheHit, SP:CacheMiss, SP:CacheInsert

SQL Server 32-bit AWE Memory

• Mapping and Un-mapping AWE memory• Mapping cost is small – equivalent to soft fault• Un-mapping cost is substantial – need to update page

tables on all processors• Pages mapped mostly 1 at a time

• Read-ahead may map multiple at a time• Perfmon:

• Buffer Manager: AWE lookup maps/sec• Pages un-mapped many at a time

• Up to 1 MB at a go• Perfmon:

• Buffer Manager: AWE unmap calls/sec• Buffer Manager: AWE unmap pages/sec

• Only DB Page Cache is able to use AWE Memory • No virtual memory pointers within

Buffer Pool & AWE

• AWE enabled (system default) for 64-bit environment

• When using AWE mechanism, buffer pool no longer uses virtual memory committed

• dbcc memorystatusMemory Manager KB ------------------------------ ----------------VM Reserved                      16979888VM Committed                  217928AWE Allocated                  14116272Reserved Memory               1024Reserved Memory In Use        0

Memory Pressure Comparison500 Pace databases, Thinktime 30sec

1000

1500

2000

2500

3000

32bit

32bit(AWE)

32bit(/3GB)

32bit(AWE/3GB)

64bit

Tpm

020406080100

CPU% TpmCPU%

500 Pace databases, Thinktime 30sec

10

15

20

25

30

32bit

32bit(AWE)

32bit(/3GB)

32bit(AWE/3GB)

64bit

Batc

h Re

q/se

c

020406080100120

SQL

com

p/se

c

BatchReq/secSQLcomp/sec

Memory Pressure Comparison…500 Pace databases, Thinktime 30sec

0.020.040.060.080.0

100.0

CPU%

, SQL

com

p/se

c

92.00

94.00

96.00

98.00

100.00

Proc

pla

ns h

it ra

tio

CPU%

SQLcomp/secProc plan hitratio

500 pace databases, Thinktime 30sec

115 135 203 234

1066

0200400600800

1,0001,200

Proc

cac

he K

-pag

es

90.0092.0094.0096.0098.00100.00

Proc

pla

ns h

it ra

tio

Proc cachepagesProc plans hitratio