23
Business Intelligence Portfolio [email protected] 908-769-9399 Manisha Menon

Manisha Menon Ssis Portfolio

Embed Size (px)

DESCRIPTION

Business Intelligence Resume - Data warehousing SSIS SSAS SSRS MDX SQL Server

Citation preview

Page 1: Manisha Menon Ssis Portfolio

Business Intelligence

Portfolio

[email protected]

Manisha Menon

Page 2: Manisha Menon Ssis Portfolio

OverviewThe project is for a fictitious construction company called AllWorks. We designed and built a SQL Server 2005 database to track employee and customer information, timesheet and labor rates data, as well as job order information, job materials, and customer invoices. This information was previously stored in Excel Spreadsheets, XML files, and .CSV files.

The objective of the SSIS project was to write ETLs to populate a SQL database named AllWorksDBStudent from source files in the form of Excel Spreadsheets, XML files, and .CSV files. The following were the packages prepared.

Employee Master Package (the starting project contains a blank version of this one)Employee Rates PackageClient Masters PackageClient Grouping PackageDivision Master PackageClient Groupings to Client Xref Master PackageProject (“Job”) Master PackageJob TimeSheets Package

Page 3: Manisha Menon Ssis Portfolio

A child package was also created to generate an email to [email protected] for the results of every package, including rows inserted and any errors generated (for example, if data loads contain child timesheet records with an invalid parent record).

A separate package was created to handle nightly database backups and another package was created to re-index all files and shrink the database.

Finally a Master Package was created to call all the above packages.The packages were scheduled to run nightly at midnight. This job was created in SQL Server Management Studio.

 

Page 4: Manisha Menon Ssis Portfolio

The EmailResultPackage.dtsx Package

This is a common package that is called from the 1st seven packages.  Cool Feature: It gets its variables about how many records were affected, PackageName, Success/Failure etc. set from the parent package using SSIS -> Package Configurations -> Parent Package Variable. Cool Feature: The email is sent from: [email protected]

Page 5: Manisha Menon Ssis Portfolio

The EmployeeMasterPackage.dtsx Package

This package inserts / updates data from C:\SetFocusBISourceData\Employees.xls, Employees sheet. It also has a child package called EmailResultPackage.dtsx to email results of running the package.

Page 6: Manisha Menon Ssis Portfolio

The EmployeeRatesPacakge.dtsx Package

Page 7: Manisha Menon Ssis Portfolio

This package inserts / updates data from C:\SetFocusBISourceData\Employees.xls, Employee Rates sheet. EmployeePK in the Employee Rates must exist in the Employee Master, else the record is not inserted into the table and the record is sent to the log file C:\SetFocusBIProjects\StudentVersion\SSISStudentProject\Logs\EmployeeRatesLog.txt . The log files are always appended to, not overwritten. It also has a child package called EmailResultPackage.dtsx to email results of running the package.

Page 8: Manisha Menon Ssis Portfolio

The ClientMasterPackage.dtsx Package

This package inserts / updates data from C:\SetFocusBISourceData\ClientGeographies.xls, Client Listing sheet.

All columns from the XL sheet are not loaded into the DB.

It also has a child package called EmailResultPackage.dtsx to email results of running the package.

Page 9: Manisha Menon Ssis Portfolio

The ClientGroupingsMasterPackage.dtsx Package

Page 10: Manisha Menon Ssis Portfolio

I have taken the liberty to design this package slightly differently – to showcase a different way of doing this task, and to learn something new. The next package which has a similar logic follows a different technique.

Cool Feature: In this package a SQL Command is run to give us the distinct rows that are needed. So right from the beginning we cut down on a lot of data being brought in. It is a good optimization technique.

This package inserts / updates data from C:\SetFocusBISourceData\ClientGeographies.xls, Special Groupings sheet. It picks the distinct ClientGroupingPK and its corresponding Name only from that sheet.  It also has a child package called EmailResultPackage.dtsx to email results of running the package.

Page 11: Manisha Menon Ssis Portfolio

The DivisionMasterPackage.dtsx Package

This package inserts / updates data from C:\SetFocusBISourceData\ClientGeographies.xls, Division Definitions sheet. It picks the distinct DivisionPK and its corresponding DivisionName only from that sheet.  It also has a child package called EmailResultPackage.dtsx to email results of running the package. Cool Feature: In this package we will use the AGGREGATE DATA FLOW ITEM TASK to achieve getting the distinct DivisionPK and its corresponding DivisionName.

Page 12: Manisha Menon Ssis Portfolio
Page 13: Manisha Menon Ssis Portfolio

The ClientGroupingsXClientsPackage.dtsx Package

Page 14: Manisha Menon Ssis Portfolio

This package inserts / updates data from C:\SetFocusBISourceData\ClientGeographies.xls, Special Groupings sheet. It is manifests a one to many relation of the ClientGrouping and Clients. The Account Key in the XLS is the ClientPK in Clients Master table.  Before insertion, the package makes sure that the ClientPK and ClientGroupingPK are valid by checking its existence in their appropriate master files. Non matching rows are sent to C:\SetFocusBIProjects\StudentVersion\SSISStudentProject\Logs\ClientGroupingsXClientLog.txt . The log files are always appended to, not overwritten. It also has a child package called EmailResultPackage.dtsx to email results of running the package.

Page 15: Manisha Menon Ssis Portfolio

The ProjectMasterPackage.dtsx Package

This package inserts / updates data from C:\SetFocusBISourceData\ProjectMaster.xls, Project Master Sheet into the Job Master Table. Project and Job are synonyms. ProjectID and JobMasterPK mean one and the same. Before insertion, the package makes sure that the ClientPK is valid by checking its existence in the Clients master file. Non matching rows are sent to C:\SetFocusBIProjects\StudentVersion\SSISStudentProject\Logs\ProjectMasterLog.txt . The log files are always appended to, not overwritten. It also has a child package called EmailResultPackage.dtsx to email results of running the package.

Page 16: Manisha Menon Ssis Portfolio

The ProjectTimesheetsPackage.dtsx Package

Page 17: Manisha Menon Ssis Portfolio
Page 18: Manisha Menon Ssis Portfolio

This package reads data from all .CSV files in the C:\SetFocusBISourceData\time directory and processes it. It expects those files to have Timesheet related data and inserts the timesheet records in the JobTimeSheets Table. Before insertion, the package makes sure that the EmployeePK is valid by checking its existence in the Employees master table. Non matching rows are sent to C:\SetFocusBIProjects\StudentVersion\SSISStudentProject\Logs\ProjectTimesheetLog.txt . The log files are always appended to, not overwritten. Before insertion, the package also makes sure that the JobMasterPK is valid by checking its existence in the JobsMaster table. Non matching rows are sent to C:\SetFocusBIProjects\StudentVersion\SSISStudentProject\Logs\ProjectTimesheetLog.txt . The log files are always appended to, not overwritten.

Page 19: Manisha Menon Ssis Portfolio

The EmployeePK, JobMasterPK and the Workdate combination must be unique for each record to be qualified for a new insert, else it will update the record. Cool Feature: It also sends email notification about the results of running the package. To do this it uses a user variable to store the intermittent status of each file processed and appends the new results to it. After all files are processed in the ForEachLoop, the email is sent with the complete message with status of processing all files, the names of the files processed, the # rows inserted / updated etc. the in the user variable.  Cool Feature: The email is sent from: [email protected]

Page 20: Manisha Menon Ssis Portfolio

The MiscTasksPackage.dtsx

This was a quick package written without much frills just to get data into the County Master.

Page 21: Manisha Menon Ssis Portfolio

The BackupDatabasePackage.dtsx

Page 22: Manisha Menon Ssis Portfolio

The ReIndexAndShrinkDatabasePackage.dtsx

Page 23: Manisha Menon Ssis Portfolio

The MasterControllerPackage.dtsx

This is the Master Controller package that calls 10 other packages which do various tasks like loading Master tables, Timesheets Table, Database Backup, ReIndexing and Shrinking DB, etc. The precedence control is set to 'On Completion' (Blue) because we want it to execute whether the called package executes with or without error as the called package is self contained and does its own error handling. Note that the sequence of calling some of the packages is important.