Upload
guillermo-caicedo
View
117
Download
6
Tags:
Embed Size (px)
DESCRIPTION
Citation preview
SSIS Tips & Tricks
Guillermo Caicedo
– Working as a Consultant at Nagnoi, LLC– MCP | SQL Server & SharePoint– President of the Puerto Rico PASS Chapter (PRPASS)– Has spoke at past events like SQL Saturday and User
Group Meetings
About Me
[email protected]@sqltitan
sqltitan.com
SSIS Tips & Tricks
Document your code!!!
– Annotations– Layout tools
– Use sequence containers to group related tasks
– Establish naming conventions for packages, components and connection managers.
Naming convention examples:
http://consultingblogs.emc.com/jamiethomson/archive/2006/01/05/SSIS_3A00_-Suggested-Best-Practices-and-naming-conventions.aspx
Task Prefix
For Loop Container FLCForeach Loop Container FELCSequence Container SEQCActiveX Script AXSAnalysis Services Execute DDL ASEAnalysis Services Processing ASPBulk Insert BLKData Flow DFTData Mining Query DMQExecute DTS 2000 Package EDPTExecute Package EPTExecute Process EPRExecute SQL SQLFile System FSYSFTP FTPMessage Queue MSMQScript SCRSend Mail SMTTransfer Database TDBTransfer Error Messages TEMTransfer Jobs TJTTransfer Logins TLTTransfer Master Stored Procedures TSPTransfer SQL Server Objects TSOWeb Service WSTWMI Data Reader WMIDWMI Event Watcher WMIEXML XML
Component Prefix
DataReader Source DR_SRCExcel Source EX_SRCFlat File Source FF_SRCOLE DB Source OLE_SRCRaw File Source RF_SRCXML Source XML_SRCAggregate AGGAudit AUDCharacter Map CHMConditional Split CSPLCopy Column CPYCData Conversion DCNVData Mining Query DMQDerived Column DERExport Column EXPCFuzzy Grouping FZGFuzzy Lookup FZLImport Column IMPCLookup LKPMerge MRGMerge Join MRGJMulticast MLTOLE DB Command CMDPercentage Sampling PSMPPivot PVTRow Count CNTRow Sampling RSMPScript Component SCRSlowly Changing Dimension SCDSort SRTTerm Extraction TEXTerm Lookup TEL
SSIS Community Tasks and Components(https://ssisctc.codeplex.com/)(http://www.sqlis.com)
BIDS Helper (https://bidshelper.codeplex.com/)
– BIMLScript– Automate your ETL Framework
Community Tools
Open Visual Studio and Skip the Splash Screen
-NOSPLASH
Change Visual Studio Theme Color
Start with a Blank Solution
Add Solution to Source Control
http://tfs.visualstudio.com
Add SSIS Project
<<Installation Directory>>:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\PrivateAssemblies\ProjectItems\DataTransformationProject\DataTransformationItems
Reuse your code - SSIS Package Templates
http://msdn.microsoft.com/en-us/library/ms345191.aspx
– Ensure consistency in terms of Coding Standards, Naming Standards, and Best Practices etc. in SSIS Packages across different Modules/Projects
– Faster Package development and testing
Change new package Name property Generate a new GUID for the ID property.
Design a strategy
Operations• Break down solution into specific units of work [Modules]
Volume• Identify data volumes at each stage
Application• Select the right SQL technology for the job
Location• Determine where the operations take place
Transformations
Synchronous Transformations - reuse existing buffers– Row transformations - manipulate data or create new
fields using the data that is available in that row
http://technet.microsoft.com/en-us/library/aa337074.aspx
Transformations
Asynchronous Transformations - a new buffer is created for the output and a new thread is introduced into the Data Flow–number of input rows will likely not match the
number of output rows–Partially blocking transformations – combine
datasets–Full Blocking transformations - must read and
process all input records before creating any output records
http://technet.microsoft.com/en-us/library/aa337074.aspx
Error Logging
Use Event HandlersSQL Server 2012 – has out of the box logging
Parallel Execution
Execute packages or tasks in parallel when possible
Tune your Data Flows
Optimize Your Query– SELECT only columns you need
– Less columns = more rows per buffer
– Consider using NOLOCK hint– Removes locking overhead– Improves the speed of large tables scan– Be Careful - can read dirty data
– Consider using OPTION (FAST 10000)– Rob Farley - http://
sqlblog.com/blogs/rob_farley/archive/2011/02/17/the-ssis-tuning-tip-that-everyone-misses.aspx
– Do as much work in the Source as possible
Bulk Inserts
Table or View – fast load at Destination
Manage your buffer size
Calculate your buffer size and adjust default settings
– DefaultMaxBufferRows: The maximum number of rows a source component will read in at any given time – Default = 10000 rows
– DefaultBufferSize: The number of bytes per buffer. – Default = 10485760 (10 MB)– Max = 104857600 (100 MB)
Fill your buffers with data as soon as possible
Fast Data Source – consider increasing the size of the buffer, more rows can be processed at the same time
Slow Data Source – consider decreasing the size of the buffer
table_size_kb 171080 kilobytes Reserved size of table in KB
table_size 175185920 bytes Size of table / 1024 bytes
row_count 4000000 rows Table total rows
buffer_width 43.79648 bytes row_count / table_size
max_buffer_size 104857600 bites DefaultBufferSize Max allowed value (100 MB)
DefaultMaxBufferRows 2394202 rows max_buffer_size / buffer_width
DefaultBufferSize 104857600 bytes max_buffer_size
Drop Non-Clustered Indexes
Dropping non-clustered indexes can improve performance.
Batch Updates
Don’t use OLE DB Command to perform updates.Insert to a temporary or staging table and execute batch updates from the Control Flow
Why?
Monitoring
Out of the Box
Community Reporting Toolshttps://ssisreportingpack.codeplex.com/
Questions? Comments?