Data-Ed Webinar: Data Warehouse Strategies

Peter Aiken, Ph.D. & Steven MacLauchlan

Data Warehousing Strategies

2

Copyright 2014 by Data Blueprint

PremiseTwo types of listeners … 1. Interested in how to

approach the subject of warehousing data – Need to integrate disparate

data – Need more holistic view of

business operations – Management just discovered

data warehouses and wants you to "build one"

2. Have complex and/or messy data warehouse practices – Want to improve them


3


1. Warehousing data in the context of data management

2. Motivation for integration technologies (reporting->BI->Analytics)

3. Warehouse integration technologies

4. Three warehousing architecture foci

5. The use of meta models

6. Guiding principles & best practices

Maslow's Hierarchiy of Needs

4


You can accomplish Advanced Data Practices without becoming proficient in the Foundational Data Management Practices however this will: • Take longer • Cost more • Deliver less • Present

greaterrisk(with thanks to Tom DeMarco)

Data Management Practices Hierarchy

Advanced Data

Practices • MDM • Mining • Big Data • Analytics • Warehousing • SOA

Foundational Data Management Practices

5


Data Platform/Architecture

Data Governance Data Quality

Data Operations

Data Management Strategy

Technologies

Capabilities

UsesReuses

What is data management?

6


Sources

Data Governance

Data Engineering

Data Delivery

DataStorage

Specialized Team Skills

Understanding the current and future data needs of an enterprise and making that data effective and efficient in supporting business activitiesAiken, P, Allen, M. D., Parker, B., Mattia, A., "Measuring Data Management's Maturity: A Community's Self-Assessment" IEEE Computer (research feature April 2007)

Data management practices connect data sources and uses in an organized and efficient manner • Storage • Engineering • Delivery • Governance

When executed, engineering, storage, and delivery implement governance

Note: does not well-depict data reuse

Maintain fit-for-purpose data, efficiently and effectively

DMM℠ Structure of 5 Integrated DM Practice Areas

7


Manage data coherently

Manage data assets professionally

Data architecture implementation

Data engineering implementation

Organizational support

Dat

a M

anag

emen

t Bod

y of

Kno

wle

dge

8


Data Management

Functions

DAMA DM BoK & CDMP

9


• Data Management Body of Knowledge (DMBOK) – Published by DAMA International, the

professional association for Data Managers (40 chapters worldwide)

– Organized around primary data management functions focused around data delivery to the organization and several environmental elements

• Certified Data Management Professional (CDMP) – Series of 3 exams by DAMA International and

ICCP – Membership in a distinct group of

fellow professionals – Recognition for specialized knowledge in a

choice of 17 specialty areas – For more information, please visit:

• www.dama.org, www.iccp.org

Data Warehousing & Business Intelligence Management

10Copyright 2014 by Data Blueprint

Warehousing data in the context of data management

11


Assumes you have • An overarching data strategy • A strategy for becoming

familiar with "big data technologies"

• Made a decision to not make available (integrating or storing) needed data

• Decided to increase (or decrease) the complexity of existing DM practices

• Decided to learn more about this DM BoK slice

UsesReusesSources

Data Governance

Data Engineering

Data Delivery

DataStorage

Specialized Team Skills


12








Payroll Application(3rd GL)Payroll Data

(database)

R& D Applications(researcher supported, no documentation)

R & D Data (raw) Mfg. Data

(home grown database)

Mfg. Applications(contractor supported)

Finance

Data (indexed)

Finance Application(3rd GL, batch

system, no source)

Marketing Application(4rd GL, query facilities, no reporting, very large)

Marketing Data

(external database)

Personnel App.(20 years old,

un-normalized data)

Personnel Data

(database)

Typical System Evolution

13


Multiple Sources of (for example)

Customer Data

Payroll Data (database)

R & D Data (raw)

Mfg. Data (home grown

database)

Finance

Data (indexed)

Marketing Data

(external database)

Personnel Data

(database)

... Then Integrate

14


OrganizationalData

Payroll Data (database)

R & D Data (raw)

Mfg. Data (home grown

database)

Finance

Data (indexed)

Marketing Data

(external database)

Personnel Data

(database)

... Then Re-architect

15


OrganizationalData

An organization's integration needs ...

16


Software Package 1

Software Package 2

Software Package 3

Software Package 4

Software Package 5

Software Package 6

Data Architecture

... map between and across software packages

Defining Data Warehousing, BI/Analytics


• Data Warehousing – A technology solution supporting … business capabilities

such as: query, analysis, reporting and development of these capabilities

– Analysis of information not previously integrated – Another, often new, set of organizational capabilities

• Business Intelligence (aka. decision support) – Dates at least to 1958 – Support better business decision making – Technologies, applications and practices for the collection, integration, analysis, and

presentation of business information – Understanding historical patterns in data to improve future performance – Use of mathematics in business

• Analytics (aka.) enterprise decision management, marketing analytics, predictive science, strategy science, credit risk analysis. fraud analytics - often based on computational modeling

• Reframing the question … – From: what data warehouse should we build? – To: how can data warehouse-based integration address challenges?

From The DAMA Guide to the Data Management Body of Knowledge © 2009 by DAMA International

Descriptive

Ask: What happened? What is happening?

Find: Structured data

Show: Profiles, Bar/pie charts, Narrative

Predictive Ask: What will happen? Why will it happen?

Find: Structured/unstructured data

Show: Risk Profiles, Pros/Cons, Care Recs

Prescriptive Ask: What should I do? Why should I do it?

Find: Unstructured/structured data

Show: Strategic Goals, Support Recs

Hemophilia Management Analytics

18


Target Isn't Just Predicting Pregnancies

19


http://rmportal.performedia.com/node/1373 and http://www.predictiveanalyticsworld.com/patimes/target-really-predict-teens-pregnancy-inside-story/ http://rmportal.performedia.com/rm/paw10/gallery_01#1373

Basics

20


• Users can "drill" anywhere

• Entire collection "cube" is accessible

• Summaries to transaction-level detail

Sample questions …

21


Cancer patient revenue across all facilities

Revenue for diseases this year versus last year in the NE region

Total costs and revenue at top 10 facilities

• Emphasis on the "cube"

– N dimensions

• Permits different users to "slice and dice" subsets of data

• Viewing from different perspectives

Example: Set Analysis


Portfolio Analysis


• Bank accounts are of varying value and risk

• Cube by – Social status – Geographical location – Net value, etc.

• Strategy or goal: balance return on the loan with risk of default

• How to evaluate the portfolio as a whole? – Least risk loan may be to the very wealthy, but there are a very

limited number – Many poor customers, but greater risk

• Solution may combine types of analyses – When to lend, interest rate charged

15 years ago, CarMax started as a way to make the car buying experience simple, fair, and fun. Today CarMax is a FORTUNE 500 retailer and one of FORTUNE’s “100 Best Companies to Work For.” And we are hiring talented individuals who are interested in:--solving original, wide-ranging, and open-ended business problems--not only discovering new insights, but successfully implementing them--making a significant mark on a growing company--developing the fundamental skills for a rewarding business careerIf that sounds like you, the Strategy Analyst position is the unique opportunity you’ve been looking for. The strategy team at CarMax currently consists of over 40 analysts, many of whom are recent college graduates from top schools with a variety of academic backgrounds (computer science, economics, English, engineering, journalism, math, political science). These analysts lead advances and decisions in several key business areas:-Inventory and pricing—what is the optimal selection of inventory, how do we acquire it, what should we pay for it, what should we price it for?-Expansion planning—which markets should we enter and how do we store those markets? Will each $10-30 million store investment generate a sufficient economic return?-Credit strategy—how can our bank (CarMax Auto Finance) approve more customers for loans and convert more approvals to sales?-Marketing and consumer insight—how do we reach our customers, increase traffic to our stores, and best use the internet to drive sales and build our brand-Industry and competitive research—what middle- and long-term risks are we exposed to, and how best do we prepare to respond? -Production—how do we increase vehicle reconditioning quality while reducing cost and production time?-Sales process and workforce—what is the best way to serve customers in our stores, and how do we manage, motivate and compensate our sales team?Even early in your career at CarMax, you will have the responsibility to own an area of the business and will be expected to improve it. For example, one undergraduate recruit used data analysis to reformulate our retail pricing strategy, pitched and sold his idea to the senior executive team, and implemented a new system nationwide in his first 6 months with the company. That is the kind of impact you can make at CarMax. And as you do this, you will work closely with the senior executives and analytical managers to develop the fundamental and advanced skills that underpin a successful career in business. In fact, most of our managers in the strategy group started at CarMax as analysts, and our VP of Strategy and Analysis started his career here through our undergraduate recruiting program. While an MBA is not required to advance or contribute at CarMax, analysts who have chosen to pursue a business degree have enjoyed superior acceptance rates at their first choice schools, including Harvard, Chicago, UVa, Columbia, and Duke.

Your opportunities to develop, contribute, and lead as an analyst at CarMax are as great as the company’s opportunity to grow. While CarMax is already the largest used car retailer in the country (with over $8 billion in sales and over 90 superstores across the country), we have only 2% of the 1 to 6-year-old used car market, which, at $280 billion annually, is bigger than the home improvement or consumer electronics industries. CarMax is already growing at 15% a year, and over the next 10 years plans to have 250-300 stores and achieve $25+ billion in annual sales. As an analyst, you can be an integral part of that growth, all while enjoying a casual and friendly environment, a diverse group of talented associates, a healthy work-life balance, and excellent compensation and benefits. An ideal candidate will have --Demonstrated top caliber analytic and problem solving skills --History of achievement demonstrated by top 15% GPA, with a quantitative major(s), and/or other recognition such as scholarships, awards, honor societies -- Passion for business and desire to develop into a strong business leader We encourage you to apply. For more information, please visit us at the career fair, on our website (www.carmax.com/collegerecruiting), or email us at [email protected]

- datablueprint.com

CarMax Example Job Posting


24

own an area of the business and will be expected to improve it

--solving original, wide-ranging, and open-ended business problems--not only discovering new insights, but successfully implementing them--making a significant mark on a growing company--developing the fundamental skills for a rewarding business career

Polling Question #1


• Do you have/have you started data warehousing, marts and/or other warehousing forms of integration? a. Last year (2014) b. This year (2015) c. Next Year (2016) d. Nope


26








Technology from The DAMA Guide to the Data Management Body of Knowledge © 2009 by DAMA International


• ETL • Change Management Tools • Data Modeling Tools • Data Profiling Tools • Data Cleansing Tools • Data Integration Tools • Reference Data Management Applications • Master Data Management Applications • Process Modeling Tools • Meta-data Repositories • Business Process and Rule Engines

Warehousing Definitions


• Inmon: – "A subject oriented, integrated, time variant, and non-volatile

collection of summary and detailed historical data used to support the strategic decision-making processes of the organization."

• Kimball: – "A copy of transaction data specifically structured for query and

analysis."

• Key concepts focus on: – Subjects

– Transactions

– Non-volatility

– Restructuring

Courtesy of: http://www.infosys.com/industries/healthcare/industryofferings/Pages/healthcare - data-warehousing.aspx

Warehousing applied to a specific challenge

29


Oracle

30


from The DAMA Guide to the Data Management Body of Knowledge © 2009 by DAMA International

Corporate Information Factory Architecture



MetaMatrix Integration Example

32


• EII Enterprise Information Integration

– between ETL and EAI - delivers tailored views of information to users at the time that it is required

Linked Data

33


Linked Data is about using the Web to connect related data that wasn't previously linked, or using the Web to lower the barriers to linking data currently linked using other methods. More specifically, Wikipedia defines Linked Data as "a term used to describe a recommended best practice for exposing, sharing, and connecting pieces of data, information, and knowledge on the Semantic Web using URIs and RDF."

linkeddata.org

Health Care Provider Data Warehouse


"A roomful of MBAs can accomplish this

analysis faster!"

• 1.8 million members • 1.4 million providers • 800,000 providers no key • 29% prov_ssn ≠ 9 digits • 2.2% prov_number = 9 digits (required) • 1 User • $30 million

Indiana Jones: Raiders Of The Lost Ark


Causes of Data Warehouse Failure


1. The project is over budget 2. Slipped schedule 3. Unimplemented functions

and capabilities 4. Unhappy users 5. Unacceptable performance 6. Poor availability 7. Inability to expand 8. Poor quality data/reports 9. Too complicated for users 10. Project not cost justified 11. Poor quality data 12. Many more values of gender code than (M/F) 13. Incorrectly structured data 14. Provides correct answer to wrong question 15. Bad warehouse design 16. Overly complex

from The Data Administration Newsletter, www.tdan.com and The DAMA Guide to the Data Management Body of Knowledge © 2009 by DAMA International

Reframing the question

37


• From: How shall we build this data warehouse? – (Worse) … What should go into this warehouse?

• To: How can warehousing capabilities solve this specific business challenge? – (Better still) … How can warehousing capabilities

solve this class of business challenges? • Other examples

– Are you ready for a data warehouse? ✓ Foundational practices

– Will you get it right the first time? ✓ Is the business environment constantly evolving? ✓ Do you have an agreed upon enterprise-wide vocabulary?

– Is your data warehouse intended to be the enterprise audit-able system of record? ✓ Extract, transform and load requirements ✓ Data transformation requirements

– How fast do you need results? ✓ Performance of inserts vs reads ✓ Project deliverables


38









Inmon Implementation/3NF

39

OPERATIONAL SYSTEM

OPERATIONAL SYSTEM

FLAT FILES

SUM MARYDATA RAW DATA

M ETADATA

PURCHASING

SALES

INVENTORY

ANALYSIS

REPORTING

MINING

Third Normal Form

40


• Each attribute in the relationship is a fact about a key

– Highly normalized structure

• Use Cases

– Transactional Systems

– Operational Data Stores

Third Normal Form: Pros and Cons

41


Neo4j.com

• Pros

– Easily understood by business and end users

– Reduced data redundancy

– Enforced referential integrity

– Indexed attributes/flexible querying

• Cons

– Joins can be expensive

– Does not scale


Kimball Implementation/Dimensional

42

• Comprised of “fact tables” that contain quantitative data, and any number of adjoining “dimension” tables

• Optimized for business reporting • Use Cases

– OLAP (Online Analytic Processing) – BI

Star Schema

43


Wikipedia

Star Schema Pros and Cons

44


• Pros

– Simple Design

– Fast Queries

– Most major DBMS are optimized for Star Schema Designs

• Cons

– Questions must be built into the design

– Data marts are often centralized on one fact table


Data Vault Implementation

45

Data Vault

46


Bukhantsov.org

• Designed to facilitate long-term historical storage, focusing on ease of implementation

• Retains data lineage information (source/date)

• “All the data, all the time” - hybrid approach of Inmon and Kimball.

• Comprised of Hubs (which contain a list of business keys that do not change often), Links (Associations/transactions between hubs), and Satellites (descriptive attributes associated with hubs and links)

• Use Cases – Data Warehousing

– Complete Audit-ability

Data Vault Pros and Cons

47


• Pros

– Simple integration

– Houses immense amounts of data with excellent performance

– Full data lineage captured

• Cons

– Complication is pushed to the “back end”

– Can be difficult to setup for many data workers

– No widespread support for ETL tools yet

Comparison

48


Polling Question #2


• Do you have? a. A single enterprise data warehouse b. Coordinated data marts c. Both d. Uncoordinated efforts e. None


50








Source:http://dmreview.com/article_sub.cfm?articleID=1000941 used with permission

Meta Data Models


Metadata Data ModelSCREENELEMENTscreen element id #data item id #screen element descr.

INTERFACEELEMENTinterface element id #data item id #interface element descr.

INPUTELEMENTinput element id #data item id #input element descr.

OUTPUTELEMENToutput element id #data item id #output element descr.

MODELVIEWmodel view element id #data item id #model view element des.

DEPENDENCYdependency elem id #data item id #process id #dependency description

CODEcode id #data item id #stored data item #code location

INFORMATIONinformation id #data item id #information descr.information request

PROCESSprocess id #data item id #process description

USER TYPEuser type id #data item id #information id #user type description

LOCATIONlocation id #information id #printout element id #process id #stored data items id #user type id #location description

PRINTOUTELEMENTprintout element id #data item id #printout element descr.

STORED DATA ITEMstored data item id #data item id #location id #stored data description

DATA ITEMdata item id #data item description

52


Warehouse Process

Warehouse Opera.on

Transforma.on

XMLRecord-‐ Oriented

Mul. Dimensional

Rela.onal

Business Informa.on

So@ware Deployment

ObjectModel (Core, Behavioral, Rela.onships, Instance)

Warehouse Management

Resources

Analysis

Object-‐ Oriented

(ObjectModel)

Foundation

OLAPData Mining

Informa.on Visualiza.on

Business Nomenclature

Data Types

ExpressionsKeys Index

Type Mapping

Overview of CWM Metamodel

http://www.omg.org/technology/documents/modeling_spec_catalog.htm


Marco & Jennings's Complete Meta Data Model

Source:http://dmreview.com/article_sub.cfm?articleID=1000941 used with permission

54



55








Guiding Principles



1. Obtain executive commitment and support 2. Secure business SMEs 3. Let the business drive the priorities 4. Demonstrate data quality is essential 5. Provide incremental value 6. Transparency and self service 7. One size does not fit all: Secure the right tools

and products for each of your segments 8. Think and architect globally, act and build locally 9. Collaborate with and integrate all other data initiatives, especially those for

data governance, data quality and metadata 10.Start with the end in mind 11.Summarize and optimize last, not first

Data Reengineering Leverage


Data Management Practices

Duplicated but ETLed Data(quality & transformations applied)

"Warehoused" Data

Learning/Feedback

Marts

Analytics Practices

58


Data Warehousing Strategies1. Warehousing data in the context of data

management






Data Warehousing & Business Intelligence Management


Questions?

60


It’s your turn! Use the chat feature or Twitter (#dataed) to submit

your questions to Peter and Steven now.

• www.datablueprint.com/webinar-schedule• www.Dataversity.net

Brought to you by:

Upcoming EventsJanuary Webinar:Developing a Data-centric Strategy & RoadmapJanuary 12, 2016 @ 2:00 PM – 3:30 PM ET(11:00 AM-12:30 PM PT)

February Webinar:The Importance of Master Data Management February 9, 2016 @ 2:00 PM – 3:30 PM ET(11:00 AM-12:30 PM PT)

Sign up here:


Appendix

62



Goals and Principles


1. To support and enableeffective business analysisand decision making byknowledgeable workers

2. To build and maintain theenvironment/infrastructure tosupport business intelligenceactivities, specificallyleveraging all the other datamanagement functions tocost effectively deliverconsistent integrated datafor all BI activities

Activities


• Understand BI information needs

• Define and maintain the DW/BI architecture

• Process data for BI

• Implement data warehouse/data marts

• Implement BI tools and user interfaces

• Monitor and tune DW processes

• Monitor and tune BI activities and performance from The DAMA Guide to the Data Management Body of Knowledge © 2009 by DAMA International

Primary Deliverables


• DW/BI Architecture

• Data warehouses, marts, cubes etc.

• Dashboards-scorecards

• Analytic applications

• Files extracts (for data mining, etc.)

• BI tools and user environments

• Data quality feedback mechanism/loop from The DAMA Guide to the Data Management Body of Knowledge © 2009 by DAMA International

Roles and Responsibilities



Suppliers:• Executives/managers• Subject Matter Experts• Data governance council• Information consumers• Data producers• Data architects/analysts

Participants: • Executives/managers• Data Stewards• Subject Matter Experts• Data Architects• Data Analysts• Application Architects• Data Governance Council• Data Providers• Other BI Professionals

Consumers:

• Application Users

• BI and ReportingUsers

• ApplicationDevelopers andArchitects

• Data integrationDevelopers andArchitects

• BI Vendors andArchitects

• Vendors, Customersand Partners

6 Best Practices for Data Warehousing


1.Do some initial architecture envisioning.

2.Model the details just in time (JIT).

3.Prove the architecture early.

4.Focus on usage.

5.Organize your work by requirements.

6.Active stakeholder participation.

http://www.agiledata.org/essays/dataWarehousingBestPractices.html

Kimball's DW Chess Pieces



5 Key Business Intelligence Trends


1. There's so much data, but too little insight. More data translates to a greater need to manage it and make it actionable.

2. Market consolidation means fewer choices for business intelligence users.

3. Business Intelligence expands from the Board Room to the front lines. Increasingly, business intelligence tools will be available at all levels of the corporation

4. The convergence of structured and unstructured data Will create better business intelligence.

5. Applications will provide new views of business intelligence data. The next generation of business intelligence applications is moving beyond the pie charts and bar charts into more visual depictions of data and trends.

http://www.cio.com/article/150450/Five_Key_Business_Intelligence_Trends_You_Need_to_Know?page=2&taxonomyId=3002













References


References


Additional References


• http://www.information-management.com/infodirect/20050909/1036703-1.html • http://www.agiledata.org/essays/dataWarehousingBestPractices.html • http://www.cio.com/article/150450/

Five_Key_Business_Intelligence_Trends_You_Need_to_Know?page=2&taxonomyId=3002 • http://www.computerworld.com/s/article/9228736/

Business_Intelligence_and_analytics_Conquering_Big_Data?taxonomyId=9 • http://www.enterpriseirregulars.com/5706/the-top-10-trends-for-2010-in-analytics-business-

intelligence-and-performance-management/ • http://www.itbusinessedge.com/cm/blogs/vizard/taking-the-analytics-pressure-off-the-data-

warehouse/?cs=50698 • http://www.informationweek.com/news/software/bi/240001922