Understanding Data Warehousing

Introduction

Introduction

Data has always been an essential ingredient to decision-making and, in modern business, the need to obtain, store, and use data has increased dramatically as the complexities and scope of the global marketplace has

expanded.

Data warehousing is an environment established for the sole purpose of gathering, integrating, and delivering

data from across multiple data sources for use in enterprise decision-making. However, its effectiveness

can be expanded to support any person, process, or system needing current and historical data which is

consistent and relatable.

Defining Data Warehouse

Data warehouse is a computing environment composed of several technologies and products, including:

– Data Acquisition– Data Management– Data Modeling– Data Quality– Data Analysis– Metadata Management– Development Tools– Storage Management– Applications– Administrative Functions

Defining Data Warehouse (Part 2)

Data Warehousing is about managing the data. The following data features are key reasons for having a data warehouse:

– Subject Orientation– Data Integration– Non-volatile– Time Variance– Data Granularity

Benefits of Data Warehousing

Data Warehousing provides the following benefits: A comprehensive and integrated perspective of the

enterprise Availability of current and historical information for

strategic decision making Mitigating operational risks related to supporting

the decision-making process Providing a flexible and interactive source of

information

Introducing Business Intelligence

Business Intelligence is a set of disciplines designed specifically to establish a consistent decision-making environment.

Business Intelligence does not replace Data Warehousing, but uses it extensively in it processes.

Business Intelligence can be described as a two-step process:

– Transforming data into information– Transforming information into knowledge

Functional Components of a Data Warehouse

Data Acquisition

Data StorageInformation

Delivery

Physical Components of a Data Warehouse

Data Sources

Data Staging

Data Storage

Information Delivery

Metadata

Source Data

Data Sources can include:– Operational Data– Internal Data – Archived Data– External Data

Data can consist of structured or unstructured, prepared or raw formats.

Data Staging

The activities of data staging are:– Extracting data from the data sources– Transforming the data into usable information– Loading the data and metadata into data storage

ETL (Data Extraction, Transformation, and Loading) is considered the most time-consuming and human-intensive activities in data warehousing.

Data Quality

One purpose of Data Staging is to raise the quality of the data used in decision making: bad data will lead to bad decisions.

Data Quality is influenced by:– Inadequate database designs– Aging of data– Dummy or absent data – Non-unique identifies– Ineffective primary keys– Violation of business rules– Lack of policies and procedures– Input errors

Data Storage

Organizations must establish the storage requirements for:

– Data staging– Corporate data warehouse– Individual data marts– OLAP-based multidimensional databases

Information Delivery

The requirements for Information Delivery reside in expectations related to:

– Query types and frequencies– Report types and frequencies– Types of analysis– Distribution of information– Real-time requirements– Applications for decision support– Potential growth and expansion

Metadata

The core of the data warehouse is its

METADATA

What is a Data Mart?

A data mart is a subset of a data warehouse. A data warehouse will typically contain data relevant to the entire enterprise, while a data mart contains data relevant to a line of business or department within the enterprise.

Deployment of data warehouses and data marts will usually take one of the following approaches:

– Top-down (data warehouse first, data marts second)– Bottom-up (data marts first, data warehouse second)

Data Warehouse Architecture

There are five basic architectures in data warehousing:

– Centralized Data Warehouse – one data warehouse with no data marts.

– Independent Data Marts – several autonomous data marts with no central data warehouse.

– Federated Data Marts – several data marts operating under standardized controls with no central warehouse.

– Hub-and-Spoke – several data marts with a central data warehouse.

– Data-Mart Bus – several data marts are created and conform to the standards and controls of the original data mart.

Why Data Warehousing?

What does a data warehouse provide the user?

– Ability to run simple queries and reports against current and historical data

– Ability to perform “what if” scenarios– Ability to iteratively query and analyze deeper into the

data– Ability to identify historical trends and apply them

effectively to future situations.

Challenges in Data Acquisition

The typical challenges facing data acquisition activities are:

– Large number of data sources– Disparate data sources– External data sources– Ongoing data feeds– Different computing platforms– Data replication– Data integration– Data cleansing– Complex data transformations

Challenges in Data Storage

The typical challenges facing data storage activities are:

– Large data volumes– Large data sets– New data types– Data storage in staging area– Multiple index types– Parallel processing– Data archiving– Tool compatibilities

Challenges in Information Delivery

The typical challenges facing information delivery activities are:

– Multiple user types– Multiple query types– Complex queries– OLAP– Multidimensional analysis– Web-enablement– Metadata management– Tools from multiple vendors

Relevant Data Warehouse Standards

Relevant standards for data warehousing, specifically metadata, are provided through:

– Meta Data Coalition– Object Management Group– OLAP Council for Multi-dimensional Application

Programmers Interface (MDAPI)

Basic Project Plan

The basic plan for a data warehouse project is:

– Planning– Defining requirements– Design– Build– Deploy– Maintain

The Toolkit

The Toolkit is designed to be holistic to the enterprise’s relationship with data, not just data warehousing. As part of its scope, a second presentation is available to introduce Data Analytics and Data Mining, which is related to the second step of Business Intelligence.

The goal of the Data Warehouse/Analytics Toolkit is to define the contributing factors, major components, and their relationships, while providing the basic tools to take action based on the organization’s needs.

Moving Forward

The participant can take two directions in using the toolkit at this point. To continue with the data warehouse discussion, the next document of interest is, Developing Warehouse Capabilities, which is intended to be a step-by-step guide in creating a Big Data foundation in your organization. To learn more about data-related activities within an enterprise, see the presentation, Introduction to Data Analytics and Mining.

. Multiple templates have been created to support the process and aid organizations in their efforts to improve their Data Warehouse and Data Analytic capabilities.