View
221
Download
0
Category
Preview:
Citation preview
Overcoming SSIS Deployment and Configuration Challenges
Erik VeermanMentor, Solid Quality MentorsSQL Server MVP
//ATLANTAMDF2
• SQL Server MVP (Most Valuable Professional)• Mentor with Solid Quality Mentors• ~15 years in the industry, 10 years SQL Server BI
experience (v1 SQL BI, beta SQL 7)… Telco, Manufacturing, Finance, Insurance, Retail, Real Estate…
• MS Worldwide BI Solution of the Year• SQL Server Magazine Innovator Cup• Microsoft Project REAL ETL Architect• Author:
• SQL Server 2008 Integration Services Professional • MS Press SQL Server BI Training Kit (70-445) (70-448 in progress!)• SQL Server 2005 Integration Services Expert• SQL Server 2005 Integration Services Professional
Speaker BIO – Erik Veerman
//ATLANTAMDF
Industry-Leading Technical Experts, Authors, Speakers
Solid Quality™ Mentors are more than 100 of the world’s top technical experts who specialize in assuring client success using integrated
Microsoft technologies.
“Catch the Solid Quality™ Advantage”
www.solidq.com
3
PRACTICE AREAS
Relational Database Management
Business Intelligence
Development Methodologies
SharePoint Collaboration
SERVICES
Advanced Public Training
Customized Private Training
Solution Delivery and Tuning
Enhanced Mentoring Services
//ATLANTAMDF4
• Who has had challenges with SSIS deployment or configurations?
• Has anyone punched a wall or other visible frustrations?
• Has anyone actually had to go to anger management?
Let’s be honest…
//ATLANTAMDF
Baby Steps: The Books-Online Basics–Security: Encryption and Connections–Configurations and Expressions–Deployment
The Problem: Bob Anger Management: The Solutions
Agenda
//ATLANTAMDF
BOL Basics: SSIS Package Security
Security is comprised of several layers to support both SQL and File System based scenarios–Packages can be encrypted–Packages can be digitally signed–Packages can be stored in SQL DB and protected with roles–Packages can be stored in the File System and protected with File
System Security
Package Definition:
Storage:
SSIS Package File (.DTSX):Encrypted, Digitally Signed
File System:File System Security
MSDB:MSDB SQL Server Roles
//ATLANTAMDF
BOL Basics: SSIS Package Security
Internal Package Security–Package encryption – Used to encrypting connection
passwords or the entire package–Package Password – Optional password applied to the
package to un-encrypt any encrypted contents External Package Security–MSDB database roles for packages deployed to SQL
Server. Reader and Writer roles included for execution and reading privileges–File system security for packages deployed to server file
systems or file shares
//ATLANTAMDF
BOL Basics: SSIS Package Encryption Levels
Don’t Save Sensitive Encrypt Sensitive with User Key Encrypt Sensitive with Password Encrypt All with User Key Encrypt All with Password Server Storage
//ATLANTAMDF
Package Encryption and MSDB
demo …
//ATLANTAMDF
BOL Basics: Securing Data and Connections
Data Security–Data is not stored as part of a package
Connection Schema Security–Connection schema information will be stored in
packages depending on the uses (Data Flow source/destinations and Execute SQL)–Lock down package access when appropriate
File system rights Package encryption/password Package roles and access
//ATLANTAMDF
Package Connections and Encryption
demo …
12
//ATLANTAMDF
SSIS Package Connection 1 = XYZ Variable 1 = XYZ
SSIS Package Connection 1 = XYZ Variable 1 = XYZ
SSIS Package Connection 1 = ABC Variable 1 = ABC
BOL Basics: SSIS Configurations
Update package properties at execution time based on external setting–Variable properties–Connection Properties–Package properties–Task and Container Properties
Data Flow Transformation properties are not configurable except at the Task level
Configuration• Connection 1 = ABC
• Variable 1 = ABC
//ATLANTAMDF
SSIS Configuration Types
XML File Configuration–Direct path or…–Environment Variable driven
SQL Server Configuration–Table Driven–Filter Column grouping configurations
Registry Configuration Parent Package Variable Configuration
–Inherit variables from master packages
Environment Variable Configuration
//ATLANTAMDF
Package Configurations
demo …
//ATLANTAMDF
BOL Basics: SSIS Property Expressions
Runtime updates of Control Flow components–Package level properties–Task and Container properties–Connections, too!
Expressions assignable through property windows and task editors
SSIS expression language based
SSIS PackageVariable1 = C:\FileA
Connection1 = Expression: @Variable1
• Step 1: Run Data Flow• Access Connection1• Extract from C:\FileA
• Step 2: Update Variable• Step 3: Run Data Flow
• Access Connection1• Extract from C:\FileB
C:\FileA
C:\FileB
C:\FileB
//ATLANTAMDF
SSIS Property Expressions
Common Property Expression usage–Dynamic SQL for non-parameterizable statements–Connections, Checkpoints, Disable flag–Loop Container enumeration functionality
Variable values can be updated by expressions through the EvaluateAsExpression and Expression property of the variable
//ATLANTAMDF
Property Expressions
demo …
//ATLANTAMDF
BOL Basics: Deployment
Deployment destinations–File System (file share or local drive)–MSDB Database (sysdtspackages90 or sysssispackages)
Deployment options–Manual windows command line script (Xcopy or Robocopy)–DTUtil command line script (file system or MSDB)–Deployment Wizard and Installer
//ATLANTAMDF
BOL Basics: Deployment
File System
MSDB
Development
Test orProduction
SSIS Package
SSIS Package
BIDS Project Folder File System or Share
File Copy, DTUtil, Installer Set
Development
Test orProduction
SSIS Package
SSIS Package
BIDS Project Folder MSDB Database
SSMS, DTUtil, Installer Set
//ATLANTAMDF
BOL Basics: Deployment Wizard
Designer can build a deployment set which includes a project's packages, configurations files and an installer file
You move the installer file set to another server/environment and execute to install packages to SQL or file system
Installer can "copy" misc files but will not install them, such as custom components
Deployment overwrites existing packages
//ATLANTAMDF
Package Deployment
demo …
//ATLANTAMDF
Baby Steps: The Books-Online Basics–Security: Encryption and Connections–Configurations and Expressions–Deployment
The Problem: Bob Anger Management: The Solutions
Agenda
24
• You want to live a normal IT life but the problem follows you around
• Just when you think you’ve solved the problem, there it is again
The Problem:
• You’re the expert, but the problem will not go away
• Just knowing the baby steps doesn’t get you far enough
//ATLANTAMDF
SSIS Deployment Challenges
Encryption causes your package to fail Connections and other properties aren’t updated Passwords are not available or too easily accessible Shared configurations require properties and connections exist SQL Configurations point to Dev database
Encryption Failure:
Server 1
Server 2
SSIS Package
Connection A
SSIS Packages MSDB or File System
SSIS Package
Connection APackage Deployed
EncryptionWith UserKey
EncryptionWith UserKey
Package Fails!
Can not decrypt
UserKey (New
Machine)
//ATLANTAMDF
SSIS Deployment Challenges
Connections and other properties aren’t updated
Server 1
Server 2
SSIS Package
Dev Connection ADev Connection B
Dev Connection A
MSDB or File System
Package Error!
Connections Point to Wrong
Databases/Files
SSIS Package
Dev Connection ADev Connection B
Package Deployed
Dev Connection B
Prod Connection A
Prod Connection B
SSIS Packages
//ATLANTAMDF
SSIS Deployment Challenges
SQL Configurations point to Dev database (or configuration database not accessible!)
Server 1
Server 2
SSIS PackageDev Connection ADev Connection BDev Config DB
Dev Connection A
MSDB or File System
Config DB Connection Not
Updated!
SSIS PackageDev Connection ADev Connection BDev Config DB
Package Deployed
Dev Connection B
Prod Connection A
Prod Connection B
Dev Config DB
1
2
1
Prod Config DB
2
SSIS Packages
//ATLANTAMDF
Pulling Your Hair Out!
demo …
//ATLANTAMDF
Baby Steps: The Books-Online Basics–Security: Encryption and Connections–Configurations and Expressions–Deployment
The Problem: Bob Anger Management: The Solutions
Agenda
Solution 1
Solution 1: Tie Bob to the Boat
• Pretend the problem is not there• Try and man-handle the problem• Make the problem worse
Solution 2Solution 2: Blow up the house
• Give up• Find a punching bag• Declare to Bill Gates that you
are not a PC anymore
//ATLANTAMDF
Dealing with Encryption
Situation: All connections allow Windows Authentication Use Windows Authentication only for all connections Turn off encryption (Set encryption to DontSaveSensitive)
Server 1
Server 2
SSIS Package
Connection A/B/C (Windows Auth)
SSIS Packages MSDB or File System
SSIS Package
Connection A/B/C (Windows Auth)
Package Deployed
Encryption:DontSave-Sensitive
Encryption:DontSave-Sensitive
//ATLANTAMDF
Dealing with Encryption
Situation: One or more connections require password Option 1: EncryptSensitiveWithPassword
Pass in the package Password during execution Option 2: Set encryption to DontSaveSensitive
Use a Configuration to update the password Option 3: Deploy to MSDB, set encryption to ServerStorage
Option 1:
Server 1
Server 2
SSIS Package
Connection A,password = ABC
SSIS Packages MSDB or File System
SSIS Package
Connection A,password = ABC
Package Deployed
Encryption:EncryptSensitive-WithPassword
Encryption:EncryptSensitive-WithPassword
Package Executed with DTExec and the /P switch
//ATLANTAMDF
Dealing with Encryption
Option 2: Set encryption to DontSaveSensitive Use a Configuration to update the password
Server 1
Server 2
SSIS Package
Connection A,password = [Blank]
SSIS Packages MSDB or File System
SSIS Package
Connection A,password = [Blank]
Package Deployed
Encryption:DontSave-Sensitive
Encryption:DontSave-Sensitive
Config File
password = ABC
Config File
password = ABC
//ATLANTAMDF
Solving the Encryption Problem
demo …
//ATLANTAMDF
Dealing with Connections Option 1: Use a File Configuration (direct or indirect path) Option 2: Use a File and SQL Configuration Option 3: Use Property Expressions and Variables
Server 1
Server 2
SSIS PackageDev Connection ADev Connection B
Dev Connection A
MSDB or File System
SSIS PackageDev Connection ADev Connection B
Package Deployed
Dev Connection B
Prod Connection A
Prod Connection B
Dev Config File
1
2
1
Prod Config File
2
SSIS Packages
Option 1
//ATLANTAMDF
Dealing with Connections Option 2: Use a File and SQL Configuration
Server 1
Server 2
SSIS PackageDev Connection ADev Connection BDev Config DB
Dev Connection A
MSDB or File System
SSIS PackageDev Connection ADev Connection BDev Config DB
Package Deployed
Dev Connection B
Prod Connection A
Prod Connection B
Dev Config File
1
3
1
Prod Config File
SSIS Packages
SSIS Dev Config DB
2
SSIS Prod Config DB2
3
//ATLANTAMDF
Demo, Two-Pass Configuration Approach
demo …
//ATLANTAMDF
Dealing with Connections Option 3: Use a Package Variable and Expression
Server 1
Server 2
SSIS Package
Connection A VariableConnection B Variable
Connection A (Expression)Connection B (Expression)
Dev Connection A
MSDB or File System
Dev Connection B
Dev Config File
1
3
SSIS Packages
24
SSIS Package
Connection A VariableConnection B Variable
Connection A (Expression)Connection B (Expression)
Dev Connection A
Dev Connection B
Dev Config File
1 2
Package Deployed
Makes sharing configurations easier!
3
4
//ATLANTAMDF
Demo, Variable Based Configuration Approach
demo …
//ATLANTAMDF
BIDS Helper
CodePlex.com – BIDS Helper Tools
–Deployment SSIS Packages tool–dtsConfig File Formatter–Expression and Configuration Highlighter–Expression List–Non-default properties list–Variables Window Extensions–FixRelativePaths–…others…
41
//ATLANTAMDF42
Consider storing your packages on SQL Server
Consider storing your packages in
the file system
Will you have hundreds of SSIS
packages
YesDo you need
Pkg execution or read privileges for a
team
Package Storage Decision Flow
Do you Need a centralized
package backup and Security
model
Are you using SQL Server RDBMS?
Are you the DBA or do you have
DBA privileges?
Can your Source Control tool handle
deployment
No
Yes
No
Yes
No
No
Will you Have a lot of Master-
Child packages?
Yes
Yes
No
Other factors: cluster, multiple
servers, etc
No
Yes
No
Yes
//ATLANTAMDF
Use DTUtil and Command Line Scripting
Do you have a set of packages to be deployed per
deployment
Package Deployment Decision Flow
Do you Need to automate
deployment
Are you storing your packages in
SQL Server?
Can your Source Control handle
deployment
Do your Packages for
deployment always come from the same
BIDS project?
No
Yes
Yes
No
No
Yes
Yes
Manually deploy
package Files
Use the Source Control
Deployment Tools for file system
deployment
Use the SSIS Deployment
Wizard
Yes
No
No
//ATLANTAMDF
Package Encryption Decision Flow (version 1)
Are you Breathing?
Yes
EncryptSensitive-WithUserKey
EncryptAll-WithUserKey
DontSave-Sensitive
EncryptSensitive-WithPassword
EncryptAll-WithPassword
ServerStorage
//ATLANTAMDF
Package Encryption Decision Flow (version 2)
Are you Obsessive
Compulsive (like Bob)?
Yes
EncryptSensitive-WithUserKey
EncryptAll-WithUserKey
DontSave-Sensitive
EncryptSensitive-WithPassword
EncryptAll-WithPassword
No
ServerStorage
//ATLANTAMDF
Package Encryption Decision Flow (version 3)
Are you the only person that will ever edit/design
the package
YesEncryptSensitive-
WithUserKey
EncryptAll-WithUserKey
DontSave-Sensitive
EncryptSensitive-WithPassword
EncryptAll-WithPassword
No
Will you always
use the same Server?
Do you need a class in
anger management
No
No
Yes
ServerStorage
Yes
//ATLANTAMDF
Package Encryption Decision Flow (version 4)
Are you deploying
to SQL Server…
And are deploying using the SSIS Deployment Tool…
And don’t use configurations…
And your passwords don’t change…
Yes
No ServerStorage
See other options…
//ATLANTAMDF
Use an indirect (Env Var to
reference) file configuration
Do you or your org have
a standard or pref. for storing settings
in files or DBs?
Package Configuration Decision Flow
Do you have passwords or
sensitive properties in the package?
Will the file name or Path change
between servers?
Yes
Tables
No
Use a direct file configurationFiles
Is your server locked down (file and DB)?
Is it easier in your org to
secure files or tables?
No
No
Yes
Use SQL Config with a File Config
as a pointer
TablesYes
Files
Other factor: Sharing configs
Thank you
Recommended