Auditing, Tracking, and Change Monitoring Technologies in Microsoft SQL Server 2008Adam MachanicDatabase ConsultantSQLblog

SESSION CODE: DAT318

About Me

Independent SQL Server and .NET ConsultantBoston, MA

Conference and INETA SpeakerConnections, PASS, TechEd, DevTeach, etc.

AuthorSQL Server 2008 InternalsExpert SQL Server 2005 Development

Co-Founder: SQLblog.comThe SQL Server Blog Spot on the Web

amachanic@gmail.com

Agenda

Change Detection: A Primer and Prehistory

Change Tracking: Which Rows Changed?

Change Data Capture: What Changed?

SQL Audit: Who Made a Change?(or: Who Read My Data?)

Agenda

Change Detection: A Primer and Prehistory

Change Tracking: Which Rows Changed?

Change Data Capture: What Changed?

SQL Audit: Who Made a Change?(or: Who Read My Data?)

Who Cares What Happened?

Your government cares.Can you produce a full disclosure report on who accessed what, and when?

Your support team cares.Metadata-driven applications? When did that setting change that caused the app to crash?

You should care.Your career just might depend on it.

Rolling Your Own: How Not to Audit

Security Auditing: SQL TraceNo easy way of telling who accessed whatCan be expensive from a resource point of viewC2 and Common Criteria modes? Close, but not quite there

Data Auditing: TriggersEasy to make a mistake, introduce bugsRequires change to database logical modelCan create performance nightmares

The Timestamp Problem

T0:Txn 1 Starts

Time (T) T1:Txn 2 Starts

T1:Txn 2 Ends

T3: App Requests Changes

T4:Txn 1 Ends

SQL Server 2008: Major Fixes

Three distinct technologies to help us figure out what happened

Microsoft v1.0 technologies: They have some pitfalls

All in all, they take us a long way towards a solid solution

Agenda

Change Detection: A Primer and Prehistory

Change Tracking: Which Rows Changed?

Change Data Capture: What Changed?

SQL Audit: Who Made a Change?(or: Who Read My Data?)

Change Tracking: OverviewAnswers the question: Which rows (keys) changed?

Does not store how the data changed

Synchronously populated during the transactionTotally integrated into the Storage Engine

Available in all SQL Server 2008 editions

Use cases:Disconnected cache updateETL update (Type 3 SCD and similar)

Change Tracking 101

demo

CT Configuration: Database-Level

ALTER DATABASE AdventureWorksSET CHANGE_TRACKING = ON;

Gets the database ready to track change

Recommend also enabling SNAPSHOT ISOLATION

Which databases are enabled? sys.change_tracking_databases

CT Configuration: Table-Level

ALTER TABLE HumanResources.EmployeeENABLE CHANGE_TRACKING;

Signals the query engine to start tracking changes for the table

Optionally, can enable tracking of which columns were referenced in an update

Change Tracking: Internal ArchitectureDatabase Commit Table

• xact_id – transaction identifier

• commit_ts – the external version #

Change Table

• xact_id• xact_sequence• xact _operation (I,U,D)• Base Table PK cols

Base Table

• xact_id – what transaction last modified the row?

(hidden column)

1: Transaction Starts

xact_id generated

2: Rows Modified

xact_id used in base and change table

3: Transaction Committed

commit_ts generated

It’s Enabled—Now What?

CHANGETABLE functionReturns version information on a per-key basis

VERSION mode Returns, per key, the most recent version number

CHANGES modeReturns, per key and original version, the net changes that have occurred since that version

Tracking Helper FunctionsCHANGE_TRACKING_MIN_VALID_VERSION

Per tableWhat is the minimum available change version?

CHANGE_TRACKING_CURRENT_VERSIONFor the databaseWhat is the current maximum change version?

CHANGE_TRACKING_CONTEXTApplied to a DML operation, allows you to store additional information along with the change

Data and Change Acquisition Workflow

SNAPSHOT Transaction

?Is MIN_VALID_VERSION

> [version from the last collection]

… or …

Does [version from the last collection] == 0

Call CHANGETABLE in VERSION mode

Get the initial snapshot

Call CHANGETABLE in CHANGES mode

Get changes since the last collection

Collect CHANGE_TRACKING_CURRENT_VERSION

Store for the next iteration

YES

NO

Change Tracking in Depth

demo

Agenda

Change Detection: A Primer and Prehistory

Change Tracking: Which Rows Changed?

Change Data Capture: What Changed?

SQL Audit: Who Made a Change?(or: Who Read My Data?)

Change Data Capture: OverviewAnswers the questions:

Which rows changed?How did the data change?

Asynchronously populated, after the transactionUses the replication log reader

Only available in SQL Server 2008 Enterprise

Use cases:All Change Tracking use cases… plus general ETL, data undo scenarios, etc

Change Data Capture 101

demo

CDC: Database-Level ConfigurationEXEC sys.sp_cdc_enable_db

Enables CDC metadata

Creates a database schema called “cdc”

Creates tables in the schema

Creates two SQL Server Agent jobsOne to fire up the log readerOne to purge history

CDC: Table-Level ConfigurationEXEC sys.sp_cdc_enable_table

Enables CDC for a given table

Creates one table in the cdc schemacdc.[schema]_[tablename]_CT

Creates two functions in the cdc schemacdc.fn_get_all_changes_[schema]_[tablename]cdc.fn_get_net_changes_[schema]_[tablename]

CDC: Job Configurationsys.sp_cdc_change_job

Change various aspects of collection and cleanup@retention – Number of minutes that a row should be kept in the change table@maxtrans and @pollinginterval – Controls how often polling is done and how many log records are read

MSDN: “Tuning the Performance of Change Data Capture in SQL Server 2008” – Steffen Krause

CDC: Helper Functions

sys.fn_cdc_get_min_lsnsys.fn_cdc_get_max_lsn

Find the minimum and maximum LSNs currently in the storeTo be used during the synchronization processSimilar in purpose to CT’s min and max functions

sys.fn_cdc_increment_lsnUsed to find the next LSN value as compared to the current stored valueUsed to determine which range to synchronize

CDC: Row Lifecycle Workflow

Step 1: Rows Modified,

Transaction Committed

Step 2:Sometime later…

Log Reader kicks in, reads the changes

from the logStep 3:

Change information written to table in

CDC schema

Change Data Capture in Depth

demo

Net Changes: CDC vs. CTSynchronicity

CT: Synchronous, on-time delivery of changesCDC: Potentially late delivery of changes

Transactional ImpactCT: 64 bytes per transaction + 26 bytes per rowCDC: 0 additional bytes per transaction

Database ImpactCT: None, after transactional impactCDC: Up to 34 bytes per row, plus ALL data

Agenda

Change Detection: A Primer and Prehistory

Change Tracking: Which Rows Changed?

Change Data Capture: What Changed?

SQL Audit: Who Made a Change?(or: Who Read My Data?)

SQL Audit: OverviewAnswers the question: Who did what, and when?

Can tell you who queried a tableCan not tell you what they actually saw

Synchronously or asynchronously populated

Available only on Enterprise Edition

Use casesGeneral security auditingDebugging

Background: Extended Events (Xevent)

Advanced tracing infrastructure—new in

SQL Server 2008

SQL Audit is built on top of XEvent

Similar scope to SQL Trace, but much bigger

surface area

Designed from the ground up for performance and

scalability

Audit Terminology

AuditA data collection sessionDefines a target, maximum queue time, maximum file size, and whether a failure should take down the service

Audit SpecificationFor which events should the audit collect data?Configured at the database or server levelEach Audit can contain many specifications: Up to one per database, one server-level

Audit Actions and Groups

Each type of event is called an “action”

Action CategoriesServerDatabaseAudit

Can work granularly or using predefined “action groups”, e.g.:SCHEMA_OBJECT_ACCESS_GROUPDATABASE_CHANGE_GROUP

Audit Data Collection

Audits can collect information in 28 different categories, depending on the action

Not all actions populate all columnsMaximum of 4000 characters of data for character data in audit records (e.g. T-SQL statement)Some actions produce multiple audit records

Multi-row records will produce a sequence_no column so that you can piece together what happened

Audit Considerations

Audit is built on top of XeventShouldn’t hurt performance too much

Sometimes you need to audit something for regulatory purposes

Exercise restraintJust because you can doesn’t mean you should

SQL Audit: Tooling!

The SQL Server Management Studio has a team actually built an interface for this feature

Creating and modifying Audits is more or less a point-and-click process

Pulling out data can (and usually should) be done with T-SQL

Tip: Don’t forget to enable both the Audit and the Audit Specification(s)

SQL Audit

demo

Summary

SQL Server 2008 provides powerful, flexible technologies to help you answer the toughest database question: What happened?

Use these features as needed, but in moderation to ensure that you don’t impact server performance

Be prepared for more and more use cases as regulatory bodies continue to work

DAT Track Scratch 2 Win

Find the DAT Track Surface Table in the Yellow Section of the TLCTry your luck to win a Zune HDSimply scratch the game pieces on the DAT Track Surface Table and Match 3 Zune HDs to win

Resources

www.microsoft.com/teched

Sessions On-Demand & Community Microsoft Certification & Training Resources

Resources for IT Professionals Resources for Developers

www.microsoft.com/learning

http://microsoft.com/technet http://microsoft.com/msdn

Learning

Complete an evaluation on CommNet and enter to win!

Sign up for Tech·Ed 2011 and save $500 starting June 8 – June 31st

http://northamerica.msteched.com/registration

You can also register at the

North America 2011 kiosk located at registrationJoin us in Atlanta next year

© 2010 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to

be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

JUNE 7-10, 2010 | NEW ORLEANS, LA