View
264
Download
4
Category
Preview:
Citation preview
5. 5. Data Flow BasicsData Flow Basics
Presented by: Pramod Singla
Pramod_singla@yahoo.co.in
.
ContentContent Recap and Q&A Data Flow Task Pipeline Architecture Data Sources
◦ Demo: ADO.NET Source ◦ Demo: Excel Source ◦ Demo: Flat File Source ◦ Demo: OLE DB Source ◦ Demo: XML Source ◦ Demo: Raw File Destination ◦ Demo: Raw File Source
Data Destinations ◦ Demo: OLE DB Destination◦ Demo: DataReader Destination◦ Demo: Excel Destination◦ Demo: Flat File Destination ◦ Demo: SQL Server Destination Analysis Services Destinations ◦ Demo: Dimension Processing ◦ Demo: Partition Processing ◦ Demo: Data Mining Model Training
Summary
Recap and Q&ARecap and Q&AVariables
◦ Demo: Introducing the Variables Window◦ Demo: Working with Variables◦ Demo :Using variables in Execute SQL task
Expressions◦ Demo: Creating Time stamped log file◦ Demo: Operators and More Variables
Expression Usage◦ Demo: Using a Variable in an Expression◦ Demo: Using an Expression with a Connection Manager◦ Demo: Using Expressions with Precedence Constraints
Data Flow Task Data Flow Task Encapsulates the data flow
engine that Extract, transform, and load data
A Data Flow task can include multiple data flows
When order is important, use multiple Data Flow tasks, each task containing one data flow.
Diagram includes the inputs, outputs, ,error outputs external columns.
Pipeline Architecture Pipeline Architecture The run-time engine implementsthe control flow and package management infrastructure
The data flow engine is a specialized, high performance engine that is exclusively dedicated to extracting, transforming, and loading data.
The data flow engine provides the in-memory buffers
Arrow in dataflow are called Service path
ADO.NET SourceADO.NET Source The ADO NET source consumes data from a .NET provider and makes the
data available to the data flows
SSIS 2012 onwards, you can use the ADO NET source to connect to Microsoft Windows Azure SQL Database
The ADO.net source provides two different data access modes for extracting data:◦ A table or view.◦ The results of an SQL statement.
Excel SourceExcel Source The Excel source extracts data
from worksheets or ranges in Microsoft Excel workbooks
The Excel source provides four different data access modes for extracting data:◦ A table or view.◦ A table or view specified in a
variable.◦ The results of an SQL statement. ◦ The results of an SQL statement
stored in a variable.
Flat File SourceFlat File Source The Flat File source reads data from a
text file. The text file can be in delimited, fixed width, or mixed format.
Delimited format uses column and row delimiters to define columns and rows.
Fixed width format uses width to define columns and rows. This format also includes a character for padding fields to their maximum width.
Ragged right format uses width to define all columns, except for the last column, which is delimited by the row delimiter.
OLE DB SourceOLE DB Source The OLE DB source extracts
data from a variety of OLE DB-compliant relational databases
The OLE DB source provides four different data access modes for extracting data:◦ A table or view.◦ A table or view specified in a
variable.◦ The results of an SQL statement. The
query can be a parameterized query.◦ The results of an SQL statement
stored in a variable.
XML SourceXML Source The XML Source does not
require a Connection Manager, but instead references a XML data source either as a◦ direct reference to a file◦ a variable containing a file reference
or ◦ a variable containing XML data.
Raw File DestinationRaw File Destination The Raw File destination writes
raw data to a file. Because the format of the data is native to the destination
write data more quickly than other destinations
You use the Raw File source to retrieve raw data that was previously written by the destination
This source does not use a connection manager.
Raw File SourceRaw File Source The Raw File source reads raw
data from a file
Reads data more quickly than other sources
Retrieve raw data that was previously written by the Raw File destination
This source does not use a connection manager.
OLEDB DestinationOLEDB Destination The OLE DB destination loads data into a variety of OLE DB-compliant databases
The OLE DB destination provides five different data access modes for loading data:
◦ A table or view. You can specify an existing table or view, or you create a new table.◦ A table or view using fast-load options. You can specify an existing table or create a new table.◦ A table or view specified in a variable.◦ A table or view specified in a variable using fast-load options.◦ The results of an SQL statement.
Does not support parameters. To execute a parameterized INSERT statement, use OLE DB Command transformation
If OLE DB destination uses fast-load data access mode, specify following options:◦ Keep identity values from the imported data file or use unique values assigned by SQL Server.◦ Retain a null value during the bulk load operation.◦ Check constraints on the target table or view during the bulk import operation.◦ Acquire a table-level lock for the duration of the bulk load operation.◦ Specify the number of rows in the batch and the commit size.
DataReader DestinationDataReader Destination Exposes the data in a data flow by using the ADO.NET DataReader interface
The data can then be consumed by other application
Excel DestinationExcel Destination The Excel destination loads data into
worksheets or ranges in Microsoft Excel workbooks
Can create new sheet based on input
In Excel, a worksheet or range is the equivalent of a table or view.
The lists of available tables in the Excel Source and Destination editors display only existing worksheets
Flat File DestinationFlat File Destination The Flat File destination writes data to a text file
This destination uses a Flat File connection manager to access the text file
Specify whether to overwrite a data in a destination file that has the same name.
SQL Server DestinationSQL Server Destination Connects to a local SQL Server database and bulk loads data
Cannot access a SQL Server database on a remote server
Users who execute packages that include the SQL Server destination require the "Create global objects" permission.
Assigning "Create Global Objects" user right to that account: Click Start, point to Administrative Tools, and then click Local Security Policy. Expand Local Policies, and then click User Rights Assignment. In the right pane, double-click Create global objects In the Local Security Policy Setting dialog box, click Add. In the Select Users or Group dialog box, click the user accounts that you want to add,
click Add, and then click OK. Click OK.
If you are running the package from either SQL Server Management Studio (SSMS) or Business Intelligence Development Studio (BIDS) or DTExecUI.exe, launch those tools under the elevated Administrator account
Dimension ProcessingDimension Processing The Dimension Processing
destination loads and processes an SQL Server Analysis Services dimension
The Dimension Processing destination includes the following features:
◦ Options to perform incremental, full, or update processing.
◦ Error configuration, to specify whether dimension processing ignores errors or stops after a specified number of errors.
◦ Mapping of input columns to columns in dimension tables.
Partition ProcessingPartition Processing The Partition Processing destination loads and processes an SQL Server Analysis
Services partition.
The Partition Processing destination includes the following features:◦ Options to perform incremental, full, or update processing.◦ Error configuration, to specify whether processing ignores errors or stops after a specified number of errors.◦ Mapping of input columns to partition columns.
SummarySummary Data Flow Task Pipeline Architecture Data Sources
◦ Demo: ADO.NET Source ◦ Demo: Excel Source ◦ Demo: Flat File Source ◦ Demo: OLE DB Source ◦ Demo: XML Source ◦ Demo: Raw File Destination ◦ Demo: Raw File Source
Data Destinations ◦ Demo: OLE DB Destination◦ Demo: DataReader Destination◦ Demo: Excel Destination◦ Demo: Flat File Destination ◦ Demo: SQL Server Destination Analysis Services Destinations ◦ Demo: Dimension Processing ◦ Demo: Partition Processing
DEMO5_DataFlowBasics.dtsx 5_DataFlowBasics_OnlinDemo.dtsx
Resources & QuestionsResources & QuestionsContact me :
◦ Pramod_singla@yahoo.co.in◦ http://pramodsingla.wordpress.com/
Microsoft Resources:◦ http://pluralsight.com/training/courses/TableOfContents?courseName=ssis-basic&highlight=stacia-misner_08-ssis-basic-debugging!stacia-misner_06-ssis-basic-expressions*2,3#08-ssis-basic-debugging◦ http://technet.microsoft.com/en-us/library/
microsoft.sqlserver.dts.pipeline.wrapper(v=sql.105).aspx◦ http://technet.microsoft.com/en-us/library/bb522498(v=sql.105).aspx◦ http://www.mssqltips.com/sqlservertip/2557/ssis-architecture-and-internals-interview-questions/◦ http://msdn.microsoft.com/en-us/library/ms141122.aspx◦ http://www.bimonkey.com/support/ssis-component-samples-list/◦ http://technet.microsoft.com/en-us/library/ms137622.aspx◦ http://support.microsoft.com/kb/2009672◦ https://www.simple-talk.com/sql/reporting-services/using-sql-server-integration-services-to-
bulk-load-data/
Recommended