DAT 360:DAT 360:DTS in SQL Server 2000 Best PracticesDTS in SQL Server 2000 Best Practices

Euan GardenEuan Garden

Group Manager, SQL ServerGroup Manager, SQL Server

Microsoft CorporationMicrosoft Corporation

Technologies CoveredTechnologies Covered

• SQL Server 2000 DTSSQL Server 2000 DTS• SQL Server 2000 SQL Server 2000 • SQL Server 2000 Analysis ServicesSQL Server 2000 Analysis Services• Transact SQL Transact SQL • ActiveX ScriptingActiveX Scripting• SQL Server Accelerator for Business SQL Server Accelerator for Business

Intelligence (SSABI)Intelligence (SSABI)

““Best Practices” AgendaBest Practices” Agenda

• Package DesignPackage Design• Data LoadingData Loading• Analysis Services ManagementAnalysis Services Management• Auditing and Error HandlingAuditing and Error Handling• Putting it all together - SSABIPutting it all together - SSABI

Package Design PracticesPackage Design Practices

Drive Packages through MetadataDrive Packages through Metadata– Dynamic Properties TaskDynamic Properties Task• Access to Property of any objectAccess to Property of any object• Can get property values from .INI, SQL, etc.Can get property values from .INI, SQL, etc.

– Global VariablesGlobal Variables• Building blocks for communicating metadataBuilding blocks for communicating metadata• Populated by the Dynamic Properties TaskPopulated by the Dynamic Properties Task

– UDL FilesUDL Files• Configurable File for storing Connection Configurable File for storing Connection

InformationInformation

Package Design PracticesPackage Design Practices

Designing Modular Parent/Child PackagesDesigning Modular Parent/Child Packages– Execute Package TaskExecute Package Task• Allows joining of TransactionAllows joining of Transaction• Supports passing of GlobalsSupports passing of Globals

– Execute through COM AutomationExecute through COM Automation• LoadfromSQLServer, LoadfromRepository, LoadfromSQLServer, LoadfromRepository,

LoadfromStorageFile methodsLoadfromStorageFile methods• Eliminates Version Affinity Eliminates Version Affinity • Programmatic access to Package objectsProgrammatic access to Package objects

Parent/Child PackagesParent/Child Packages

Parent PackageParent Package

Fact ETLFact ETLDimension Dimension ETLETL

Global Variables, Global Variables, Return CodeReturn Code

Cube Cube ProcessingProcessing

Global Variables, Global Variables, Return CodeReturn Code

Parent/Child Package(s) Parent/Child Package(s) and UDL Filesand UDL Files

demodemo

““Best Practices” AgendaBest Practices” Agenda

• Package DesignPackage Design• Data LoadingData Loading• Analysis Services ManagementAnalysis Services Management• Auditing and Error HandlingAuditing and Error Handling• Putting it all together - SSABIPutting it all together - SSABI

Data Loading PracticesData Loading Practices

• Leveraging High Perf. Interface(s)Leveraging High Perf. Interface(s)– Execute SQL Task Rowset ParameterExecute SQL Task Rowset Parameter– ADO Disconnected Recordsets ADO Disconnected Recordsets – DTS Multiphase Data PumpDTS Multiphase Data Pump

• BenefitsBenefits– Cache Rowset on Client (fast lookups)Cache Rowset on Client (fast lookups)– Hold updates on ClientHold updates on Client– Conditional Inserts/Updates w/out Data Conditional Inserts/Updates w/out Data

Driven QueryDriven Query

Conditional Inserts Conditional Inserts and Updatesand Updates

demodemo

Advanced PracticesAdvanced Practices

• ApproachesApproaches– Multi-Threaded Data LoadingMulti-Threaded Data Loading– Task LoopingTask Looping– Communication through Global VariablesCommunication through Global Variables

• BenefitsBenefits– Higher Utilization during Data LoadHigher Utilization during Data Load– Compressed Data Loading WindowCompressed Data Loading Window– Easy Orchestration within DTSEasy Orchestration within DTS

Multi-Threaded Data Multi-Threaded Data File LoadFile Load

demodemo

““Best Practices” AgendaBest Practices” Agenda

• Package DesignPackage Design• Data LoadingData Loading• Analysis Services ManagementAnalysis Services Management• Auditing and Error HandlingAuditing and Error Handling• Putting it all together - SSABIPutting it all together - SSABI

Analysis Services and DTSAnalysis Services and DTS• Analysis Services Processing TaskAnalysis Services Processing Task

– Supports Processing of all OLAP ObjectsSupports Processing of all OLAP Objects– Customizable Processing through TreeKey Customizable Processing through TreeKey

and other propertiesand other properties• Prominent PropertiesProminent Properties

– TreeKey TreeKey ServerName\DBName\CubeFolder\CubeName ServerName\DBName\CubeFolder\CubeName

– ItemType ItemType 1=OLAP DB, 4=Cube, 7=Partition and 1=OLAP DB, 4=Cube, 7=Partition and

9=Dimension 9=Dimension – ProcessingOption ProcessingOption

0=Full Process, 1=Refresh Data and 0=Full Process, 1=Refresh Data and 2=Incremental Update 2=Incremental Update

Analysis Services and DTSAnalysis Services and DTS• Decision Support ObjectsDecision Support Objects

– Provide access to the AS Management Provide access to the AS Management InterfacesInterfaces

– Automated via AxScriptAutomated via AxScript– Supports Partition ManagementSupports Partition Management• CreatingCreating• CloningCloning• DroppingDropping• Merging of partitionsMerging of partitions

– Should be an OLAP Administrator to UseShould be an OLAP Administrator to Use

DSO Object HierarchyDSO Object Hierarchy

Dynamic Partition Dynamic Partition Creation and Creation and ProcessingProcessing

demodemo

Analysis Services and DTSAnalysis Services and DTS

BenefitsBenefits– Opportunity to Dynamically Manage Opportunity to Dynamically Manage

Partitions and other ObjectsPartitions and other Objects

– Support for Dynamic ProcessingSupport for Dynamic Processing

– Inclusion in the BI Solution LoadInclusion in the BI Solution Load

““Best Practices” AgendaBest Practices” Agenda

• Package DesignPackage Design• Data LoadingData Loading• Analysis Services ManagementAnalysis Services Management• Auditing and Error HandlingAuditing and Error Handling• Putting it all together - SSABIPutting it all together - SSABI

Auditing and Error HandlingAuditing and Error Handling• Logging OptionsLogging Options

– Logging to MSDBLogging to MSDB– Logging to Event LogLogging to Event Log– Logging to Text FileLogging to Text File

• Enabled through PropertiesEnabled through Properties– Granularity at Step and Package LevelGranularity at Step and Package Level

• Enabled through Data Pump PropertiesEnabled through Data Pump Properties– Capture Erroneous DataCapture Erroneous Data

Auditing and Error HandlingAuditing and Error Handling

Scripting OptionsScripting Options• DTSPackageLog Object DTSPackageLog Object

– WriteTaskRecord()WriteTaskRecord()– WriteStringToLog()WriteStringToLog()

Lineage OptionsLineage Options• DTS Lineage Global VariablesDTS Lineage Global Variables

– DTSLineage_FullDTSLineage_Full– DTSLineage_ShortDTSLineage_Short

Custom Logging Custom Logging using Intrinsic DTS using Intrinsic DTS OptionsOptions

demodemo

Auditing and Error HandlingAuditing and Error HandlingMonitoring Execution – Simple as Writing SQLMonitoring Execution – Simple as Writing SQL

select select

package.name, package.description, package.name, package.description, step.stepname, step.stepname,

step.errorcode, step.errordescription step.errorcode, step.errordescription

fromfrom

msdb..sysdtspackagelog package, msdb..sysdtspackagelog package, msdb..sysdtssteplog stepmsdb..sysdtssteplog step

wherewhere

package.lineagefull = step.lineagefull andpackage.lineagefull = step.lineagefull and

step.errorcode <> 0step.errorcode <> 0

““Best Practices” AgendaBest Practices” Agenda

• Package DesignPackage Design• Data LoadingData Loading• Analysis Services ManagementAnalysis Services Management• Auditing and Error HandlingAuditing and Error Handling• Putting it all together - SSABIPutting it all together - SSABI

ETLM Architecture OverviewETLM Architecture Overview

SourceSourceSystemsSystems

DTSDTS

ETLETL

Subject MatterSubject MatterDbDb

StagingStagingDbDb KPI’sKPI’s

CubesCubes

*.txt

DTS

DTS

DTSDTS

Loading Staging TablesLoading Staging Tables

• Source Data RequirementsSource Data Requirements– Cleansed, validated dataCleansed, validated data– Separate data file for each tableSeparate data file for each table– Data files are | delimited Data files are | delimited

• Data Load ProcessData Load Process– DTS Package calls the BULK INSERT T-SQL StatementDTS Package calls the BULK INSERT T-SQL Statement– BULK INSERT parameters controlled by DTS Package BULK INSERT parameters controlled by DTS Package

global variablesglobal variables

StagingStagingDbDb

*.txt

1|ABC Co.|110…DTS

DTS

• Staging Data OptionsStaging Data Options– Inserts and Updates can be inferred or Inserts and Updates can be inferred or

indicatedindicated

– Slowly Changing Dimension supportSlowly Changing Dimension support

• Data Load ProcessData Load Process– DTS Package calls Stored ProceduresDTS Package calls Stored Procedures– Stored Procedures provide error Stored Procedures provide error

handlinghandling– Transform Data task uses Copy Transform Data task uses Copy

Column transformationColumn transformation

Loading Dimension and Fact Loading Dimension and Fact TablesTables

StagingStagingDbDb

DTS

DTS

Processing Dimensions and Processing Dimensions and CubesCubes

• DimensionsDimensions– Incremental vs. Full Incremental vs. Full

ProcessingProcessing

• CubesCubes– Data RefreshData Refresh– Incremental vs. Full Incremental vs. Full

ProcessingProcessing– Partitioning by MonthPartitioning by Month

DTS

DTS

KPI’sKPI’sCubesCubes

SQL Server SQL Server Accelerator for Accelerator for Business IntelligenceBusiness Intelligence

demodemo

For More InformationFor More Information• Encore DevelopmentEncore Development

www.encoredev.comwww.encoredev.com

• Professional Association for SQL Server Professional Association for SQL Server www.sqlpass.orgwww.sqlpass.org

• Professional SQL Server 2000 DTS Professional SQL Server 2000 DTS wrox press, www.wrox.comwrox press, www.wrox.com

• DTS newsgroupDTS newsgroupmicrosoft.public.sqlserver.dtsmicrosoft.public.sqlserver.dts

• SQLServer CentralSQLServer Centralwww.sqlservercentral.comwww.sqlservercentral.com

• SQLDTS Site SQLDTS Site www.sqldts.comwww.sqldts.com

• SQL Server Developers Network SQL Server Developers Network www.sqldev.netwww.sqldev.net

Questions?Questions? euang@microsoft.comeuang@microsoft.com

Don’t forget to complete the Don’t forget to complete the on-line Session Feedback form on-line Session Feedback form on the Attendee Web siteon the Attendee Web site

https://web.mseventseurope.com/teched/https://web.mseventseurope.com/teched/

© 2002 Microsoft Corporation. All rights reserved.© 2002 Microsoft Corporation. All rights reserved.This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.