SQLintersection keynote a tale of two teams

High Performance and High Availability with

SQL Server 2012 AlwaysOnSumeet Bansal, Fusion-IO

Kevin Kline, SQL Sentry

2© SQLintersection. All rights reserved.

http://www.SQLintersection.com

IntroductionA Tale of Two Teams

Two rival teams …… each working to satisfy an important customer.

What’s the hardware solution? What’s the software solution?



The Customer

A real-world major financial institution headquartered in London. A core banking application - credit card transactions from ATM and

Branches Requirement: 10,000 Business Transactions / sec (Not IOPs!) Highly available using AlwaysOn across hundreds of nodes in many

Availability Groups

... AND IT LOOKS LIKE THIS AT LOAD...

o



Meeting the Requirements: Hardware

High Performance on SQL Server means tuning the FULL STACK.

Key Takeaway: This is NOT going to be easy…

OS

SQL

CPU

HBA

NIC

Array

Cache

Spindles

http://thefutureofthings.com/upload/items_icons/X-25E-Intel-Enterprise-Clas_large.jpg



First Surprise - Memory

At scale, SQL Server does a generally good job of memory management by default.

Some improvements are possible on large CPU/Memory boxes dedicated to SQL Server:

Lock Pages in Memory Big performance gain! Use gpedit.msc to grant it to SQL Service account

Large page Allocations (-TF834) On Windows 2008R2 previous issues with this TF are fixed Around 10% throughput increase

NUMA node memory distribution: Beware! Set max memory close to box max if dedicated box

available

o



Second Surprise - NICs

At scale, network traffic will generate a LOT of interrupts for the CPU

These must be handled by CPU Cores Must distribute packets to cores for processing

Rule of thumb (OTLP): 1 NIC / 16 Cores Watch the DPC activity in Taskmanager Remove SQL Server (using affinity masking) from the NIC

cores

o



Drive Selection - General Number of files matter for SQL Server

TempDB and user database has multiple files, on segregated arrays

Other important configs: NTFS allocation size at 64-KB; HBA queue depth at 64; Storport HBA

Driver

Number of drives matter More drives = more speed True for both SAN and DAS ... Less so for SSD, but still relevant (especially for NAND)

If designing for performance, make sure the topology can handle it!

Understand the path to the drives Consider workload: Random or Sequential?

Key Takeaway: Validate and compare configurations prior to deployment



Rules of Thumb – Disk IO Traditional Spindle

throughput 10K RPM – 100-130 IOPs, ‘full

stroke’ 15K RPM – 150-180 IOPs, ‘full

stroke’ Can achieve 2x or more when

‘short stroking’ the disks (using less than 20% capacity of the physical spindle)

These are for random 8K I/O Aggregate throughput

when sequential access: Between 90MB/sec and

125MB/sec for a single drive If truly sequential, any block

size over 8K will give you these numbers

Some 3.5” drives slightly faster than 2.5”

Approximate latency: 3-5ms

Cable speed Theoretical: 1.5GB/sec Typical: 1.2GB/sec

PCI-e v2 Bus X4 slot: 1.5 – 1.8GB/sec X8 slot: 3GB/sec

HBA speed 4Gbit – ~500MB/sec 8Gbit – ~1GB/sec on PCI-e X4

v2 bus Typical: 350-400MB/sec on

4Gbit, doubled on 8Gbit

u



What’s Causing these Non-Disk Bottlenecks?

Added disk pair

Backplane limit

140

140

110

Added controller



Understand the Full Stack to the drives

Key Takeaway: The deeper the topology, the greater latency, the more important the tuning

Best Practices: Understand topology, potential bottlenecks and

theoretical throughput of each component in the path! Engage storage engineers early in the process

Two major topologies for SQL Server Storage

DAS – Direct Attached Storage SAN – Storage Area Networks



Traditional Centralized Architecture

11

Application CPU and Memory HBA Switches Target

AdaptersCPU and Memory

RAID Controllers HDD/SSD

SERVERS

Active and Archive Data

STORAGE (Performance Optimized)NETWORK

Milliseconds

DatabasesVirtualizationWeb-scale

Latency and Processing Time



Shared Data Decentralization

12

Active Data

Application CPUs NAND Flash Raid Controller HDD/SSD

SERVERS

Latency and Processing Time

Archive Data

Microseconds Milliseconds

DatabasesVirtualizationWeb-scale



The SAN – Panacea to All IO Issues…

….YEAH RIGHT!

Green: Checkpoint, Red: tx/sec, Black: Disk Latency

o



DAS vs. SAN - Summary

Feature SAN DASCost High, offset by better

utilizationLow, may waste space

Flexibility More, abstraction allows online configuration changes

Less, get it right the first time!

Skills required Complex with steep learning curve

Simple and well understood

Additional Features

Snapshots; Storage Replication; Thin Provisioning

None

Performance Not high performance technology

High performance for small investment

Reliability More, very high reliability Less, depending on RAID level

Clustering Support

Yes No (special implementations exist)

So, which should we choose?

SAN DAS

o



Let’s See What It Can Do!

1 x MS SQL Server 25 Billion Transactions/Day

(Equivalent to the number of estimated Credit card transactions around the globe in a single day)

http://www.fusionio.com/blog/powering-global-commerce-with-sql-server-iomemory/

4 x 1.2TB

http://www.fusionio.com/blog/powering-global-commerce-with-sql-server-iomemory/

Demo

Turn difficult disk IO tuning into easyioMemory plug-n-play.



Meeting the Requirements: Software

Highly transparent instrumentation means monitoring the FULL STACK.

Key Takeaway: This is NOT going to be easy…

OS

SQL

CPU

HBA

NIC

Array

Cache

Spindles

ov




Instrumentation: PerfMon

Throughput: Measured in MB/sec or IOPs by PerfMon: Logical Disk Disk Read Bytes / Sec Disk Write Bytes / Sec Disk Read / Sec Disk Writes / Sec

Latency: Measured in milliseconds (ms) by PerfMon: Logical Disk Avg. Disk Sec / read Avg. Disk Sec / write

More on healthy latency values later Key Takeway: For transparency, PerfMon gives a limited picture of

performance.

o



Instrumentation: Profiler / Trace

High overhead Lots of experience needed to filter the results Deprecated! (But only for relational engine).

Key Takeway: Shows triggered events, but not a comprehensive view of whole system. Not a reliable long-term solution.

o



Instrumentation: DMVs

-- SQL Server 2012 Diagnostic Information Queries, by Glenn Berry, @GlennAlanBerry

-- http://sqlserverperformance.wordpress.com/-- http://sqlskills.com/blogs/glenn/

-- Get total buffer used by DB for current instance

SELECT DB_NAME(database_id) AS [Database Name], COUNT(*) * 8/1024.0 AS [Cached Size (MB)]FROM sys.dm_os_buffer_descriptors WITH (NOLOCK)WHERE database_id > 4 -- system databases AND database_id <> 32767 -- ResourceDBGROUP BY DB_NAME(database_id)ORDER BY [Cached Size (MB)] DESC OPTION

(RECOMPILE);

Great information! Built in for SQL Server 2005+. No history. No correlation. No interpretation.

Key Takeway: Very useful. Not very useable.

o



Instrumentation: Extended Events

Low overhead Lots of experience needed to filter the results How much memory or space? Other administrative questions to

answer…

Key Takeway: Deep data, but is it actionable and proactive information?

o



Instrumentation: Notifications

Per server setup Requires SQLAgent service Can only capture error msg/lvl, WMI metrics, PerfMon metrics

Key Takeway: Alerts are available, but high support requirements and limited proactivity.

o

Demo

Bringing all the instrumentation together for meaningful, actionable performance information.

ow



Meeting the Requirements: HA

Need more flexibility than in legacy approaches like log shipping and database mirroring.

Need a shared nothing architecture.

Key Takeaway: This is not too bad UNTIL we scale up …

OS

SQL

CPU

HBA

NIC

Array

Cache

Spindles




Availability Groups Fundamentals

o



Special Considerations: AlwaysOn

Granular control and some visibility into AlwaysOn through SSms Rt-Click Show Dashboard.

Designed for small scale implementations. As with earlier tools, user carries the risk and requirement for expertise.

o

Demo

HA + DR management and monitoring at scale.

ox



How Did We Do It?OS

SQL

CPU

HBA

NIC

Array

Cache

Spindles

OS +SQL

CPU

Fusion-io + SQL Sentry




References

Thomas Kejser, SQLCAT, and high performance IO tuning: http://blog.kejser.org/tag/sqlcat/ http://blog.kejser.org/

Jonathan Kehayias & xEvents: http://www.sqlskills.com/blogs/jonathan/category/extended-eve

nts/

Joe Sack & AlwaysOn: http://www.sqlskills.com/blogs/joe/answering-questions-with-th

e-alwayson-dashboard/

SQLPerformance.com (Jonathan Kehayias) instrumentation overhead analysis:

http://www.sqlperformance.com/2012/10/sql-trace/observer-overhead-trace-extended-events

http://blog.kejser.org/tag/sqlcat/

http://blog.kejser.org/tag/sqlcat/

http://blog.kejser.org/

http://blog.kejser.org/

http://www.sqlskills.com/blogs/jonathan/category/extended-events/



http://www.sqlskills.com/blogs/joe/answering-questions-with-the-alwayson-dashboard/








Review

High performance IO is very hard when restricted to disk-only architectures.

ioMemory from Fusion-IO is the solution! Highly transparent monitoring and alerting,

especially for HA, is very hard with native tools and features.

Performance Advisor from SQL Sentry is the solution!

Visit our booths to see the latest releases and sign up for free trials and demonstrations!

Don’t forget to enter your evaluation of this session using EventBoard!

Questions?

Thank you!

Technology

SQLintersection keynote a tale of two teams