Data Warehousing

ISYS 650

What is a data warehouse?• A data warehouse is a subject-oriented, integrated,

nonvolatile, time-variant collection of data in support of management’s decision.

– Subject-oriented: data is organized around major subjects of the enterprise, such as sales, rather than individual transactions, and is oriented to decision making.

– Integrated: the same piece of information collected from various systems is referred to in only one way.

• Example: Gender: M, F; Male, Female; Sex: 0, 1– Nonvolatile: Data is loaded into a data warehouse on a

scheduled basis.– Time-variant: Historical data to support time-series and

trend analysis.

What is a Data Warehouse?• A physical repository where relational data are

specially organized to provide enterprise-wide, cleansed data in a standardized format

• “The data warehouse is a collection of integrated, subject-oriented databases designed to support DSS functions, where each unit of data is non-volatile and relevant to some moment in time”

Need for Data Warehousing• Separation of operational and informational systems and

data for improved performance.

Types of Data in a DW

• Current detailed data: consistent at the time the data is extracted from the transaction system.

• Old detailed data: need to be archived.• Summarized data• Metadata:

– A directory of what is in the warehouse.– A guide to mapping data from transaction

database to data warehouse

Data MartA departmental data warehouse that stores only relevant data

– Dependent data mart A subset that is created directly from a data warehouse

– Independent data martA small data warehouse designed for a strategic business unit or a department

DW Framework

DataSources

ERP

Legacy

POS

OtherOLTP/wEB

External data

Select

Transform

Extract

Integrate

Load

ETL Process

EnterpriseData warehouse

Metadata

Replication

A P

I

/ M

iddl

ewar

e Data/text mining

Custom builtapplications

OLAP,Dashboard,Web

RoutineBusinessReporting

Applications(Visualization)

Data mart(Engineering)

Data mart(Marketing)

Data mart(Finance)

Data mart(...)

Access

No data marts option

Extraction, transformation, and load (ETL)

Data Integration and the Extraction, Transformation, and Load (ETL) Process

Packaged application

Legacy system

Other internal applications

Transient data source

Extract Transform Cleanse Load

Datawarehouse

Data mart

Representation of Data in DW

• Dimensional Modeling – a retrieval-based system that supports high-volume query access

• Star schema – the most commonly used and the simplest style of dimensional modeling– Contain a fact table surrounded by and connected to several

dimension tables– Fact table contains the descriptive attributes (numerical values)

needed to perform decision analysis and query reporting– Dimension tables contain classification and aggregation information

about the values in the fact table

• Snowflakes schema – an extension of star schema where the diagram resembles a snowflake in shape

Multidimensionality

• MultidimensionalityThe ability to organize, present, and analyze data by several dimensions, such as sales by region, by product, by salesperson, and by time (four dimensions)

• Multidimensional presentation – Dimensions: products, salespeople, market segments, business units,

geographical locations, distribution channels, country, or industry– Measures: money, sales volume, head count, inventory profit, actual

versus forecast– Time: daily, weekly, monthly, quarterly, or yearly

Example: Northwind Database

Examples of Sales Analysis

• Total sales by Product• Sales related to Customer:

– Location: Sales by City, Country

• Sales related to Time:– Quarterly, monthly, yearly Sales

• Sales related to Employee:

Analyze Sales DataDetailed Business Data

• Total sales:• Amount of each detail line:

Quantity*UnitPrice*Discount• Sum (Quantity*UnitPrice*Discount)

• Total quantity sold:• Sum(Quantity)

• Detailed business data: – Quantity*UnitPrice*Discount– Quantity

Dimensions for Data Analysis:Factors relevant to the detailed business data

• Analyze sales by:– Product, product category– Location: City, State, Country– Time:

• Quarterly, yearly sales

– Employee:– And combinations of these dimensions:

• Ex: Product and Location, Product and Time

Data Warehouse Design- Star Schema -

• Dimension tables– contain descriptions about the subjects of the

business such as customers, employees, locations, products, time periods, etc.

• Fact table– contain detailed business data with links to

dimension tables.

Define Product Dimension

• Product Table:– ProductID, ProductName, SupplierID, CategoryID,

QuantityPerUnit, UnitPrice, UnitsInStock, UnitsOnOrder, ReorderLevel, Discontinued

• Product dimension table:– ProductID, ProductName, CategoryID

Define Employee Dimenstion

• Employees Table: EmployeeID, LastName, FirstName, Title, TitleOfCourtesy, BirthDate, HireDate, Address, City, Region, PostalCode, Country, HomePhone, etc.

• Employee Dimension:– EmployeeID, FullName, Title, EmpCity

Define Location Dimension• Customers table:

– CustomerID, CompanyName, ContactName, ContactTitle, Address, City, Region, PostalCode, Country, Phone, Fax

• Location dimension:– LocationCode, City, Country– Define Location Code: This is an artificial code created

to link detailed business data with the city and country.– In the Northwind database, I used the Make Table

query to create a Location table from the Customers table with City and Country fields. Then I used the Customers table’s design view to add a LocationCode field with the Auto Number data type.

Define Period Dimension • Period:

– In the Orders table: OrderDate– In the data warehouse we define Period to be:

PeriodCode, Year, Quarter• OrderDate: 04-Jul-1996 -> 1996, 3, 7• OrderDate: 20-Dec-1996 -> 1996, 4, 12

– In Access: Create view based on Orders table• Year:Year(OrderDate); Month:Month(OrderDate)• Quarter:

– Quarter: IIf([month]<=3,1,IIf([month]<=6,2,IIf([month]<=9,3,4)))

– Define Period Code: • PeriodCode:Cstr(Year) + Cstr(Quarter)• 1996, 3, 7 -> 19963• 1996, 4, 12 -> 19964

Star Schema

FactTableLocationCodePeriodCodeEmployeeIDProductID

QtyAmount

LocationDimension

LocationCodeCity

Country

EmployeeDimensionEmployeeID

FullNameTitle

EmpCity

ProductDimensionProductID

ProductNameCategoryID

PeriodDimensionPeriodCode

YearQuarter

A Query to retrieve data for Fact Table

Transfer Data Between Access Databases

• Create/Query/Design View– 1. Create the query with the data to transfer– 2. Click Make Table button

• Make table in the same database• Make table in other database

– 3. Click Run