SQL Server Integration Services Tips & Tricks

SSIS Tips & Tricks

Guillermo Caicedo

– Working as a Consultant at Nagnoi, LLC– MCP | SQL Server & SharePoint– President of the Puerto Rico PASS Chapter (PRPASS)– Has spoke at past events like SQL Saturday and User

Group Meetings

About Me

guillermo.caicedo@prpass.org@sqltitan

sqltitan.com

SSIS Tips & Tricks

Document your code!!!

– Annotations– Layout tools

– Use sequence containers to group related tasks

– Establish naming conventions for packages, components and connection managers.

Naming convention examples:

http://consultingblogs.emc.com/jamiethomson/archive/2006/01/05/SSIS_3A00_-Suggested-Best-Practices-and-naming-conventions.aspx

Task Prefix

For Loop Container FLCForeach Loop Container FELCSequence Container SEQCActiveX Script AXSAnalysis Services Execute DDL ASEAnalysis Services Processing ASPBulk Insert BLKData Flow DFTData Mining Query DMQExecute DTS 2000 Package EDPTExecute Package EPTExecute Process EPRExecute SQL SQLFile System FSYSFTP FTPMessage Queue MSMQScript SCRSend Mail SMTTransfer Database TDBTransfer Error Messages TEMTransfer Jobs TJTTransfer Logins TLTTransfer Master Stored Procedures TSPTransfer SQL Server Objects TSOWeb Service WSTWMI Data Reader WMIDWMI Event Watcher WMIEXML XML

Component Prefix

DataReader Source DR_SRCExcel Source EX_SRCFlat File Source FF_SRCOLE DB Source OLE_SRCRaw File Source RF_SRCXML Source XML_SRCAggregate AGGAudit AUDCharacter Map CHMConditional Split CSPLCopy Column CPYCData Conversion DCNVData Mining Query DMQDerived Column DERExport Column EXPCFuzzy Grouping FZGFuzzy Lookup FZLImport Column IMPCLookup LKPMerge MRGMerge Join MRGJMulticast MLTOLE DB Command CMDPercentage Sampling PSMPPivot PVTRow Count CNTRow Sampling RSMPScript Component SCRSlowly Changing Dimension SCDSort SRTTerm Extraction TEXTerm Lookup TEL

SSIS Community Tasks and Components(https://ssisctc.codeplex.com/)(http://www.sqlis.com)

BIDS Helper (https://bidshelper.codeplex.com/)

– BIMLScript– Automate your ETL Framework

Community Tools

Open Visual Studio and Skip the Splash Screen

-NOSPLASH

Change Visual Studio Theme Color

Start with a Blank Solution

Add Solution to Source Control

http://tfs.visualstudio.com

Add SSIS Project

<<Installation Directory>>:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\PrivateAssemblies\ProjectItems\DataTransformationProject\DataTransformationItems

Reuse your code - SSIS Package Templates

http://msdn.microsoft.com/en-us/library/ms345191.aspx

– Ensure consistency in terms of Coding Standards, Naming Standards, and Best Practices etc. in SSIS Packages across different Modules/Projects

– Faster Package development and testing

Change new package Name property Generate a new GUID for the ID property.

Design a strategy

Operations• Break down solution into specific units of work [Modules]

Volume• Identify data volumes at each stage

Application• Select the right SQL technology for the job

Location• Determine where the operations take place

Transformations

Synchronous Transformations - reuse existing buffers– Row transformations - manipulate data or create new

fields using the data that is available in that row

http://technet.microsoft.com/en-us/library/aa337074.aspx

Transformations

Asynchronous Transformations - a new buffer is created for the output and a new thread is introduced into the Data Flow–number of input rows will likely not match the

number of output rows–Partially blocking transformations – combine

datasets–Full Blocking transformations - must read and

process all input records before creating any output records

http://technet.microsoft.com/en-us/library/aa337074.aspx

Error Logging

Use Event HandlersSQL Server 2012 – has out of the box logging

Parallel Execution

Execute packages or tasks in parallel when possible

Tune your Data Flows

Optimize Your Query– SELECT only columns you need

– Less columns = more rows per buffer

– Consider using NOLOCK hint– Removes locking overhead– Improves the speed of large tables scan– Be Careful - can read dirty data

– Consider using OPTION (FAST 10000)– Rob Farley - http://

sqlblog.com/blogs/rob_farley/archive/2011/02/17/the-ssis-tuning-tip-that-everyone-misses.aspx

– Do as much work in the Source as possible

Bulk Inserts

Table or View – fast load at Destination

Manage your buffer size

Calculate your buffer size and adjust default settings

– DefaultMaxBufferRows: The maximum number of rows a source component will read in at any given time – Default = 10000 rows

– DefaultBufferSize: The number of bytes per buffer. – Default = 10485760 (10 MB)– Max = 104857600 (100 MB)

Fill your buffers with data as soon as possible

Fast Data Source – consider increasing the size of the buffer, more rows can be processed at the same time

Slow Data Source – consider decreasing the size of the buffer

table_size_kb 171080 kilobytes Reserved size of table in KB

table_size 175185920 bytes Size of table / 1024 bytes

row_count 4000000 rows Table total rows

buffer_width 43.79648 bytes row_count / table_size

max_buffer_size 104857600 bites DefaultBufferSize Max allowed value (100 MB)

DefaultMaxBufferRows 2394202 rows max_buffer_size / buffer_width

DefaultBufferSize 104857600 bytes max_buffer_size

Drop Non-Clustered Indexes

Dropping non-clustered indexes can improve performance.

Batch Updates

Don’t use OLE DB Command to perform updates.Insert to a temporary or staging table and execute batch updates from the Control Flow

Why?

Monitoring

Out of the Box

Community Reporting Toolshttps://ssisreportingpack.codeplex.com/

Questions? Comments?