Migrating your On Premise database to Azure SQL DW · 0ljudwlqj\rxu 2q 3uhplvhgdwdedvh wrdq $]xuh64/ 'dwdzduhkrxvh (uzlq gh .uhxn %xvlqhvv $qdo\wlfv &orxg $ufklwhfw _ 6shdnhu

$Page 1: Migrating your On Premise database to Azure SQL DW · 0ljudwlqj\rxu 2q 3uhplvhgdwdedvh wrdq $]xuh64/ 'dwdzduhkrxvh (uzlq gh .uhxn %xvlqhvv $qdo\wlfv &orxg $ufklwhfw _ 6shdnhu$
Migrating yourOn Premise database to anAzure SQL Datawarehouse

Erwin de KreukBusiness Analytics/Cloud Architect | Speaker

30-05-2018

2

WHO I AM

Erwin de Kreuk

+31 627 07 65 20

[email protected]

Axians since 2008 | Business Analytics/Cloud Architect | Speaker

https://nl.linkedin.com/in/erwindekreuk

@erwindekreuk


EXPO SPONSORS

Event Sponsors

Expo Light Sponsors

AGENDA

What is an Azure SQL Datawarehouse

Migration Options

Implementation

Lessons Learned

Questions

WHAT IS AN AZURE DATAWAREHOUSE

AZURE SQL DATABASE VERSUS AZURE SQL DW

AZURE SQL DATABASE AZURE SQL DW

Size Maximum 4 TB per DB Up to 1 Petabyte with compression (240TB on disk)

Concurrent Queries Up to 6400 Up to 128

Active Connections Up to 32000 Up to 1024

Cross Database Queries Supported Not Supported

Pause>Resume Not Supported Supported

Scalability Verticale Scale14 (Tiers) (Basic/Standard/Premium)

Horizontal Scale12 Options(with slider)

Replication Supported through secondarydatabase in different regions(up to 4)

Not Supported

Polybase Not Supported Supported

In Memory OLTP Supported Not Supported

Cache Data Not Supported Not Supported

Resource Classes Not Supported Static

Migrating your On Premise database to an Azure SQL DW

AZURE SQL DATABASE AZURE SQL DW GEN 1 AZURE SQL DW GEN 2

Size Maximum 4 TB per DB Up to 1 Petabyte with compression (240TB on disk)

Up to 1 Petabyte with compression (240TB on disk)

Concurrent Queries Up to 6400 Up to 128 Up to 128

Active Connections Up to 32000 Up to 1024 Up to 1024

Cross Database Queries Supported Not Supported Not Supported

Pause>Resume Not Supported Supported Supported

Scalability Verticale Scale14 (Tiers) (Basic/Standard/Premium)



Replication Supported through secondarydatabase in different regions(up to 4)

Not Supported Not Supported

Polybase Not Supported Supported Supported

In Memory OLTP Supported Not Supported Not Supported

Cache Data Not Supported Not Supported Supported

Resource Classes Not Supported Static Dynamic/Static

WHAT IS AN AZURE SQL DATAWAREHOUSE (GEN2)?


WHAT IS AN AZURE SQL DATAWAREHOUSE?

PAAS Solution in Azure

Massive Parallel Processing System

Distributed Storage

Scalable Compute (DWU Gen1 or cDWU Gen2)


AZURE SQL DW | DATAWAREHOUSE UNIT (DWU)


Combination of CPU, memory and I/O bandwidth per compute node

Heap will load the data the fastest way and can be slower to read fromClustered ColumnStore Index will load the data slower but is faster to read from

Loading in a Gen2 environment will be faster due to built-in adaptive caching that automatically caches data based on workload characteristics and query access

220.975.137 rows (view based on 3 tables)Writing the result back to a new table

500 DWU

Clustered ColumnStore Index 250 seconds

Clustered ColumnStore IndexWith hashKey

260 seconds

HEAP(NO INDEX) 230 seconds

500 DWU 1000 DWU

Clustered ColumnStore Index 250 seconds 185 seconds


260 seconds 120 seconds

HEAP(NO INDEX) 230 seconds 80 seconds

500 DWU 1000 DWU 2000 DWU

Clustered ColumnStore Index 250 seconds 185 seconds 70 seconds


260 seconds 120 seconds 85 seconds

HEAP(NO INDEX) 230 seconds 80 seconds 65 seconds

AZURE SQL DW | SCALABLE COMPUTE

Scale compute up or down when needed (saving cost)


Scale compute up or down when needed(saving cost)Scale between 100 DWU and 6000 DWU for GEN1

• Dividing DWU by 100 gives you the number of Compute Nodes• 60 Distribution Nodes

• 1000 DWU => 10 Compute Nodes• 3000 DWU => 30 Compute Nodes• 6000 DWU => 60 Compute Nodes



Performance level

Compute nodes

Distributions per Compute

node

Memory per data

warehouse (GB)

DW100 1 60 24DW200 2 30 48DW300 3 20 72DW400 4 15 96DW500 5 12 120DW600 6 10 144DW1000 10 6 240DW1200 12 5 288DW1500 15 4 360DW2000 20 3 480DW3000 30 2 720DW6000 60 1 1440



• 1000 DWU => 10 Compute Nodes• 3000 DWU => 30 Compute Nodes• 6000 DWU => 60 Compute Nodes

Scale between 1000 cDWU and 30000 cDWU for GEN2• Dividing DWU by 500 gives you the number of Compute Nodes

• 1000 cDWU => 2 Compute Nodes• 10000 cDWU => 20 Compute Nodes• 30000 cDWU => 60 Compute Nodes



Performance level

Compute

nodes

Distributionsper Compute

node

Memory per data warehouse

(GB)DW1000c 2 30 600DW1500c 3 20 900DW2000c 4 15 1200DW2500c 5 12 1500

DW3000c 6 10 1800DW5000c 10 6 3000DW6000c 12 5 3600DW7500c 15 4 4500DW10000c 20 3 6000

DW15000c 30 2 9000DW30000c 60 1 18000




Possibility to Pause Compute for Maximum Cost Savings






Possibility to Pause Compute for Maximum Cost SavingsManage compute power through:

• Azure portal






Possibility to Pause Compute for Maximum Cost SavingsManage compute power through:

• Azure portal• PowerShell

• Set-AzureRmSqlDatabase -DatabaseName "MySQLDW" -ServerName "MyServer" -RequestedServiceObjectiveName "DW1000“• Rest API

• PATCH https://management.azure.com/subscriptions/{subscription-id}/resourceGroups/{resource-group-name}/providers/Microsoft.Sql/servers/{server-name}/databases/{database-name}?api-version=2014-04-01-preview HTTP/1.1Content-Type: application/json; charset=UTF-8{

"properties": {"requestedServiceObjectiveName": DW1000

}}

• TSQL• ALTER DATABASE MySQLDW

MODIFY (SERVICE_OBJECTIVE = 'DW1000’)Migrating your On Premise database to an Azure SQL DW


AZURE SQL DW | TABLE DISTRIBUTIONS

A MPP System has different table options compared to a standard SQL DBTable Distribution Options

• Round_Robin• Is the default option• Distribute data evenly but randomly• Loads the data as fast as possible• Will incur more date movement at query time

• Hash Distributed• Distribute data based on hashing values from a single column(not an identity column)• Designed to provide high query performance on large tables• Possibility to re-distribute the data on a different column in a later stadium

• Replicated Tables• Full copy of the table accessible on each Compute node• Replicating a table removes the need to transfer data among Compute nodes before a join or aggregation• Works best for smaller tables

Temporary Tables• Table starts with #• Distributed options as mentioned above are also available• Only visible to the session in which they were created• Automatically dropped when that session is ended• Temporary tables offer a performance benefit because their results are written to local rather than remote storage


WHY CHOOSE FOR AZURE SQL DW

Scalable Growth for UsePerformant Easy to add additional computing capacity so that charging times and

usability remain optimal for usersSafe By default databases are maintained for 7 days. Snapshot are created

every 4 to 8 hoursAvailability Always available 99.9% of the time guaranteedAlways up to date Environment is automatically updated with updates. Latest updates are

available first in the cloudFast In combination with incremental loading processes or polybase, it is very

suitable for connecting all source systemsFuture proof Easy to extend with more unstructured data sources and data science

applicationsHosted in Europe Choice for cloud hosted in the Netherlands or IrelandWorld-recognized Azure cloud is used worldwide


MIGRATION OPTIONS

MIGRATION OPTIONS


Datawarehouse Migration Utility;

Data Factory;

Data Platform Studio (REDGATE);

Can be downloaded from:https://docs.microsoft.com/en-us/azure/sql-data-warehouse/sql-data-warehouse-migrate-migration-utility

MIGRATION OPTIONS | DATAWAREHOUSE MIGRATION UTILITY

DATAWAREHOUSE MIGRATION UTILITY


Can be used to migrate Schema and Data


Schema• Option to choose Distribution Round Robin or Hash• Create script needs to be executed manually in SSMS

CREATE TABLE [dbo].[HUB_PREFERENCE] ([XI_H_PREFERENCE_PKEY] Int NOT NULL,[PREFERENCE_AK] VarChar(20) NULL,[XI_REC_START_LDTS] DateTime NULL,[XI_REC_SRC] VarChar(255) NULL,[XI_IAUDITKEY] Int NULL) WITH (CLUSTERED COLUMNSTORE INDEX, DISTRIBUTION = ROUND_ROBIN)

CREATE TABLE [dbo].[HUB_PREFERENCE] ([XI_H_PREFERENCE_PKEY] Int NOT NULL,[PREFERENCE_AK] VarChar(20) NULL,[XI_REC_START_LDTS] DateTime NULL,[XI_REC_SRC] VarChar(255) NULL,[XI_IAUDITKEY] Int NULL) WITH (CLUSTERED COLUMNSTORE INDEX, DISTRIBUTION = HASH([XI_H_PREFERENCE_PKEY]))



Data• Migration Wizard is using BCP• Important to define a proper DELIMITER

Generate Scripts• Data is Exported to Local Disk

EXPORT:bcp "select TOP 100 PERCENT * from [DATABASENAME].[dbo].[HUB_PREFERENCE]" queryout "dbo.HUB_PREFERENCE.txt" -S “SERVERNAME" -T -r "\r" -t "|$|$|" -c

IMPORTbcp “Azure_DWName.dbo.HUB_PREFERENCE" in "dbo.HUB_PREFERENCE.txt" -S "azureLogicalServerName" -U “UserName@ azureLogicalServerName " -P “PassWord" -r "\r" -t "|$|$|" -q -e "dbo.HUB_PREFERENCE.log" -c


MIGRATION OPTIONS | DATA FACTORY

MIGRATION OPTIONS | DATA FACTORYAnnouncedLast week

during INFA18

DATA FACTORY



DATA FACTORY

Schema• Schema will be created during the mapping proces and depends if a table exists or not• Table is created by default with Distribution = Round_Robin


DATA FACTORY

Data• Data is transferred into a Blob Storage• Data is inserted from a Blob Storage in a SQL DW (Polybase)• Data is removed from a Blob Storage after the data has been inserted


DATA FACTORY

Fault Tolerance


DATA FACTORY

Monitoring


https://beta.dataplatformstudio.com/

MIGRATION OPTIONS | DATA PLATFORM STUDIO(REDGATE)

DATA PLATFORM STUDIO (REDGATE)




Schema• Schema will be created during the mapping proces and existing tables will be overwritten• Table is created by default with Distribution = Round_Robin



Data• Data is transferred into a Blob Storage, split up in small files and compressed



Data• Data is transferred into a Blob Storage, split up in small files and compressed• Data is loaded from External Tables in a SQL DW (Polybase)• External tables are not removed. Data resides in Blob Storage. Easy to reload the data again for testing



Monitoring


MIGRATION OPTIONS | DIFFERENCES

DWH Migration Utility DataFactory Data Platform Studio

Logging YES YES YES

Gateway needed NO YES YES

Fault Tolerance NO YES NO

Monitoring NO YES YES

Local diskspace required YES NO NO

Blob Storage required NO YES YES

Distribution Keys YES NO NO

Easy to Reload data NO NO YES

Identity Insert NO NO NO

Restartable Proces NO YES YESMigrating your On Premise database to an Azure SQL DW

IMPLEMENTATION


IMPLEMENTATION

Based on Data Platform Studio (REDGATE)

Current Situation• Migration of Staging database (350 tables) • Migration of DataVault database (700 tables, db size 1,5 TB)• Migration of Interface database(420 Business Views based on DataVault Database object)• Migration of SSIS Packages created with BIML designed with Axians Data Integration Accelerator• Current Server 64 Cores SQL Server Enterprise 512 gb Memory

Migrated Situation• 1 Azure SQL Datawarehouse with Staging, DataVault en Business View objects• Redesign a pragmatic way all SSIS Packages created with BIML


IMPLEMENTATION

We have created a SQL Datawarehouse of 200 DWU, a Blob Storage and ADF V2(SSIS)

Above steps are the minimal requirements to start a migration, the more DWU how faster themigration will be

Started the Migration Wizard from the DataPlatform Studio to transfer the Objects

StagingDatabase

DataVaultDatabase

InterfaceDatabase

Recreate viewsRemove database namesReplacing unsupported Functionality (f.e. Try_convert)


IMPLEMENTATION

After Migrating, all tables are created with Distribution=Round_Robin

HUB SATELITE

CREATE TABLE [dbo].[HUB_PREFERENCE]([XI_H_PREFERENCE_PKEY] [INT] NOT NULL,[PREFERENCE_AK] [VARCHAR](20) NULL,[XI_REC_START_LDTS] [DATETIME] NULL,[XI_REC_SRC] [VARCHAR](255) NULL,[XI_IAUDITKEY] [INT] NULL)WITH(DISTRIBUTION = ROUND_ROBIN,CLUSTERED COLUMNSTORE INDEX)

CREATE TABLE [dbo].[SAT_PREFERENCE]([XI_S_PREFERENCE_PKEY] [INT] NULL,[XI_H_PREFERENCE_PKEY] [INT] NULL,[NAM_NAME] [VARCHAR](200) NULL,[COD_EXTERNAL_CODE] [VARCHAR](20) NULL,[NAM_EXTERNAL_NAME] [VARCHAR](50) NULL,[BIT_IS_PREFERENCE] [BIT] NULL,[BIT_IS_ESSENTIAL] [BIT] NULL,[BIT_IS_MEDICAL] [BIT] NULL,[BIT_IS_PERSONAL] [BIT] NULL,[BIT_IS_SERVICE] [BIT] NULL,[XI_REC_START_LDTS] [DATETIME] NULL,[XI_REC_END_LDTS] [DATETIME] NULL,[XI_REC_SRC] [VARCHAR](255) NULL,[XI_IAUDITKEY] [INT] NULL,[XI_HASH] [VARCHAR](32) NULL)WITH(DISTRIBUTION = ROUND_ROBIN,CLUSTERED COLUMNSTORE INDEX)


IMPLEMENTATION

Changing tables for query performance and to minimize changes to the current SSIS Packages;

Hub Add Distribution Key (previous PK) CREATE TABLE dbo.HUB_PREFERENCE_PKEYWITH ( CLUSTERED COLUMNSTORE INDEX , DISTRIBUTION = HASH(XI_H_PREFERENCE_PKEY)) AS SELECT * FROM dbo.HUB_PREFERENCEOPTION (LABEL =' CTAS : dbo.HUB_PREFERENCE_HEAP_PKEYhash');drop table dbo.HUB_PREFERENCERENAME OBJECT dbo.HUB_PREFERENCE_PKEYhash TO HUB_PREFERENCE;

Satelite Add Distribution key(previous FK) CREATE TABLE dbo.SAT_PREFERENCE_PKEYhashWITH ( CLUSTERED COLUMNSTORE INDEX , DISTRIBUTION = HASH(XI_H_PREFERENCE_PKEY)) AS SELECT * FROM dbo.SAT_PREFERENCEOPTION (LABEL =' CTAS : dbo.SAT_PREFERENCE_PKEYhash');drop table dbo.SAT_PREFERENCERENAME OBJECT dbo.SAT_PREFERENCE_PKEYhash TO SAT_PREFERENCE;

Satelite Add IDENTITY column ALTER TABLE dbo.SAT_PREFERENCE

drop column XI_S_PREFERENCE_PKEYGOALTER TABLE dbo.SAT_PREFERENCEadd XI_S_PREFERENCE_PKEY int IDENTITY(1,1) NOT NULL


IMPLEMENTATION

Creating an “Identity column” for a Hub_Object• To avoid reprogramming in the first stage we have created a way to insert an Identity Column• Changed the code in BIML so we could recreate all SSIS Packages in a couple of minutes

INSERT INTO [dbo].[HUB_PREFERENCE] (XI_H_PREFERENCE_PKEY,PREFERENCE_AK,XI_REC_START_LDTS,XI_REC_SRC,XI_IAUDITKEY)

SELECT mx.MAX_pkey + ROW_NUMBER() OVER (ORDER BY mx.MAX_pkey) ,PREFERENCE_AK,“loaddate" AS XI_REC_START_LDTS,'DAVINCI' AS XI_REC_SRC,0 AS XI_IAUDITKEY

FROM ( SELECT DISTINCT SOURCE_SCHEMA.SOURCE_PREFERENCE.PREFERENCE_ID AS PREFERENCE_AKFROM SOURCE_SCHEMA.SOURCE_PREFERENCE) SRCCROSS JOIN ( SELECT MAX(XI_H_PREFERENCE_PKEY) AS MAX_pkey

FROM [dbo].[HUB_PREFERENCE]) mxWHERE NOT EXISTS ( SELECT 1

FROM [dbo].[HUB_PREFERENCE] h WITH (NOLOCK)WHERE (SRC.PREFERENCE_AK = h.PREFERENCE_AK));


IMPLEMENTATION

Avoid using Lookups in SSIS• Azure SQL DW will give you the best performance when the query runs in the

Engine

Add joins in Retrieve Data Query (Changed code in BIML)• 1 of the reasons to add staging objects to SQL DW

(to avoid cross database Joins)


IMPLEMENTATION

Statistics • Statistics help to create the most optimal plan for your queries• Process of creating and updating Statistics was a manual process• Since 3 weeks Auto create/update statistics are available


IMPLEMENTATION

Recreating Defaults on Columns;• All defaults were lost during the migration process• We have designed a script which creates an

output for all defaults which can be executed

WITH CTE_TABLEAS (SELECT TABLE_SCHEMA,

TABLE_NAMEFROM INFORMATION_SCHEMA.TABLES

WHERE TABLE_CATALOG = 'DATABASE'AND TABLE_SCHEMA = 'dbo'AND SUBSTRING(TABLE_NAME, 1, 3) IN ( 'SAT' )AND TABLE_NAME NOT LIKE '%_ext'),

CTE_PKEY_COLUMNAS (SELECT TABLE_SCHEMA,

TABLE_NAME,COLUMN_NAME,ORDINAL_POSITION

FROM INFORMATION_SCHEMA.COLUMNSWHERE TABLE_CATALOG = 'DATABASE'

AND TABLE_SCHEMA = 'dbo'AND SUBSTRING(TABLE_NAME, 1, 3) IN ( 'SAT' )AND COLUMN_NAME LIKE '%_REC_END_%'AND TABLE_NAME NOT LIKE '%_ext')

SELECT DISTINCT '---CHANGING TABLE ' + CTE_TABLE.TABLE_NAME + CHAR(10) +CHAR(10) + 'ALTER TABLE '

+ CTE_TABLE.TABLE_SCHEMA + '.' + CTE_TABLE.TABLE_NAME +CHAR(10) + CHAR(10)

+ 'ADD DEFAULT (''9999.12.31'') FOR ' + COLUMN_NAME + CHAR(10) +CHAR(10) + 'GO' AS AddDefaultEndDateFROM CTE_TABLELEFT OUTER JOIN CTE_PKEY_COLUMN

ON CTE_TABLE.TABLE_SCHEMA = CTE_PKEY_COLUMN.TABLE_SCHEMAAND CTE_TABLE.TABLE_NAME = CTE_PKEY_COLUMN.TABLE_NAME

WHERE 1 = 1AND SUBSTRING(CTE_TABLE.TABLE_NAME, 1, 3) IN ( 'SAT' );

OUTPUT:

ALTER TABLE dbo.SAT_PREFERENCE

ADD DEFAULT ('9999.12.31') FOR XI_REC_END_LDTS


IMPLEMENTATION

Start using CTAS (Create Table AS) instead of using the SSIS Dataflow or Select Into• Select .. Into, does not allow you to change either the distribution method or the index type as part of the

operationIt is a fully parallelized operation that creates a new table based on the output of a SELECT statementYou can use CTAS to create a workaround for unsupported Features

• Like ANSI JOINS on UPDATEs, ANSI JOINs on DELETEs or MERGE statement


UPDATEs

SET s.XI_REC_END_LDTS = DATEADD(ss, -1, x.max_date)FROM [dbo].[SAT_PREFERENCE] sCROSS APPLY (

SELECT MAX(XI_REC_START_LDTS) AS max_date FROM [dbo].[SAT_PREFERENCE] WHERE XI_H_PREFERENCE_PKEY = s.XI_H_PREFERENCE_PKEY

) xWHEREs.XI_REC_START_LDTS < x.max_dateAND s.XI_REC_END_LDTS = '9999/12/31';

-- Create an interim tablecreate table ctas_SAT_PREFERENCE WITH (DISTRIBUTION = hash(XI_H_PREFERENCE_PKEY)) asSELECT MAX(XI_REC_START_LDTS) max_date, XI_H_PREFERENCE_PKEY FROM [dbo].[SAT_PREFERENCE] WHERE XI_REC_END_LDTS = '9999/12/31' GROUP BY XI_H_PREFERENCE_PKEYHAVING COUNT(*) > 1;-- Use an implicit join to perform the update

UPDATE [dbo].[SAT_PREFERENCE] SET XI_REC_END_LDTS = DATEADD(ss, -1, ctas.max_date)FROM ctas_SAT_PREFERENCE ctasWHERE XI_REC_START_LDTS < ctas.max_dateAND XI_REC_END_LDTS = '9999/12/31'AND [dbo].[SAT_PREFERENCE].XI_H_PREFERENCE_PKEY = ctas.XI_H_PREFERENCE_PKEY;

-- Drop the interim tableDROP TABLE ctas_SAT_PREFERENCE

READING DATA FROM A SQL DW TO A AZURE SQL DB

Using Parameters in a SSIS DataFlow task is not Supported



Using Parameters in a SSIS DataFlow task is not SupportedEasy workaround for this is to create Execute SQL Task (EST) in the Control Flow

• Define the Parameters in Execute SQL Task• Use a CTAS to write the result to a “preload” table

• Add the parameters you wanted to add

• Run the DataFlow from the “preload” table



Elastic Query with SQL Data Warehouse

Connection between Azure SQL DB and a Azure SQL DW

Create an external Table in Azure SQL DB which directly point to a Table in AzureSQL DW

A recursive querie is not supported in an Azure SQL DW, but running the query as anElastic one you can know achieve this, because you run the query on the Azure SQL DB Site


MONITORING A SQL DW

Azure Portal


MONITORING A SQL DW

Using DMVs


SELECT * FROM sys.dm_pdw_request_stepsWHERE request_id = 'QID####' ORDER BYstep_index;

MONITORING A SQL DW

Using SQL Operations, by creating Widgets


LESSONS LEARNED

Add all tables (Staging/DataVault/Views) in the same database, cross database queries are not allowed

Use CTAS to load data as fast as possible

Define a decent distribution key• Easy to change afterwards

Define Replicate tables

Create and update Statistics

Do not pause or scale your SQL DW if long running queries are running• It will take a while to finish and you cannot stop it

Start migrating the pragmatic way, start optimizing afterwards


THINK BIG START SMALL BUT DO START!

QUESTIONS


Thank you for attention

Documents

Migrating your On Premise database to Azure SQL DW · 0ljudwlqj\rxu 2q 3uhplvhgdwdedvh wrdq $]xuh64/ 'dwdzduhkrxvh (uzlq gh .uhxn %xvlqhvv $qdo\wlfv &orxg $ufklwhfw _ 6shdnhu