CHANGE DATA CAPTURE - Amazon Web Services · •Stop the capture and cleanup jobs •Change the schema as necessary •Generate new capture instances for all modified tables •Process

CHANGE DATA CAPTURE

Charles Hyman

www.NetComLearning.com

CHANGE DATA CAPTURE

• Turn it on

EXECUTE sys.sp_cdc_enable_db;

• Which generates for the whole database

• A capture job

• A cleanup job

• Then include at least one table with

EXECUTE sys.sp_cdc_enable_table …;

• Which creates a capture instance containing

• A change table

• An all-changes function

• A net-changes function (optional)

transactions

warehouse

sources

changes

capture

job

archive

sectorETL

analysis

data

mart

ETL


CAPTURE INSTANCE

• Contains

• The capture table

• cdc.fn_cdc_get_all_changes_<instance name> function

• cdc.fn_cdc_get_net_changes_<instance name> function (optional)

• Maximum two per source table


SYS.SP_CDC_ENABLE_TABLE

• @source_schema

• @source_name

• @supports_net_changes

• @role_name

• @filegroup_name

• @index_name

• @captured_column_list

• @allow_partition_switch

• @capture_instance


DEMONSTRATION

Enabling Change Data Capture (CDC)

Creating a CDC Capture Instance

Querying CDC Tables as Things Change


SQL SERVER AGENT JOBS

• cdc.Capture_capture

• Starts with SQL Agent

• RAISERROR(22801, 10, -1);

• EXEC sys.sp_MScdc_capture_job;

• Just a wrapper for sys.sp_cdc_scan

• cdc.Capture_cleanup

• Runs at 02:00 daily

• EXEC sys.sp_MScdc_cleanup_job;


CDC DATA RETENTION

• Based on LSN Validity Intervals

• Database

• Capture Instance

• The cleanup job deletes CDC data acording to retention policy

SELECT * FROM msdb.dbo.cdc_jobs WHERE job_type = N'cleanup';

• You can start the cleanup job manually

EXEC sys.sp_cdc_start_job @job_type = N'cleanup';


HANDLING SCHEMA CHANGES

• Deleted and new columns are handled well

• Modified columns require specific steps:

• Stop the capture and cleanup jobs

• Change the schema as necessary

• Generate new capture instances for all modified tables

• Process all data in the old and new capture instances

• Manually run the cleanup job

• Delete the old capture instance

• Turn the capture and cleanup jobs back on


CHANGE TRACKING

• Turn it on for one database

ALTER DATABASE <DBNAME> SET

CHANGE_TRACKING = ON;

• Then enable a table

ALTER TABLE <TBLNAME> ENABLE

CHANGE_TRACKING;

transactionswarehouse

sources archive

sectorETL

sync


CHANGE TRACKING TABLES & FUNCTIONS

• sys.change_tracking_databases

• sys.change_tracking_tables

• CHANGETABLE(CHANGES)

• CHANGETABLE(VERSION)

• CHANGE_TRACKING_CURRENT_VERSION()

• CHANGE_TRACKING_MIN_VALID_VERSION()

• CHANGE_TRACKING_IS_COLUMN_IN_MASK()

• WITH CHANGE_TRACKING_CONTEXT


DEMONSTRATION

Enabling Change Tracking (CT) for a Database

Enabling CT on a Table

Querying CT Tables and Functions as Things Change


HANDLING SCHEMA CHANGES

• No modifications to the primary key are allowed including related indexes

• Dropping columns is OK but they may still appear in the change data

• When adding columns, changes are tracked but the metadata change is not reported

• Switching partitions will fail on change tracked changes

• Data type changes are not tracked


WHICH ONE IS RIGHT FOR ME?

Change Data Capture

• Works in Enterprise Edition only

• Stores every discrete change

• Storage intensive

• Good for auditing

• Requires SQL Server agent

• No special serialization required

Change Tracking

• Works in all versions of SQL Server

• Returns differences from current

• Storage light

• Good for device synchronization

• No job agent required

• Operates best with snapshot isolation


Documents

CHANGE DATA CAPTURE - Amazon Web Services · •Stop the capture and cleanup jobs •Change the schema as necessary •Generate new capture instances for all modified tables •Process