Data-Ed: Data Systems Integration & Business Value Pt. 3: Warehousing

Copyright 2013 by Data Blueprint

Data Systems Integration & Business Value Part 3: WarehousingCertain systems are more data focused than others. Usually their primary focus is on accomplishing integration of disparate data. In these cases, failure is most often attributable to the adoption of a single pillar (silver bullet). The three webinars in the Data Systems Integration and Business Value series are designed to illustrate that good systems development more often depends on at least three DM disciplines (pie wedges) in order to provide a solid foundation. Integrating data across systems has been a perpetual challenge. Unfortunately, the current technology-focused solutions have not helped IT to improve its dismal project success statistics. Data warehouses, BI implementations, and general analytical efforts achieve the same levels of success as other IT projects – approximately 1/3rd are considered successes when measured against price, schedule, or functionality objectives. The first step is determining the appropriate analysis approach to the data system integration challenge. The second step is understanding the strengths and weaknesses of various approaches. Turns out that proper analysis at this stage makes actual technology selection far more accurate. Only when these are accomplished can proper matching between problem and capabilities be achieved as the third step and true business value be delivered.

Date: September 10, 2013Time: 2:00 PM ET/11:00 AM PTPresenter: Peter Aiken, Ph.D.

1


Commonly Asked Questions

1) Will I get copies of the slides after the event?

2) Is this being recorded so I can view it afterwards?

2


Get Social With Us!

Live Twitter FeedJoin the conversation!

Follow us: @datablueprint

@paikenAsk questions and submit your

comments: #dataed

Like Us on Facebookwww.facebook.com/datablueprint

Post questions and commentsFind industry news, insightful

content and event updates.

Join the GroupData Management & Business

IntelligenceAsk questions, gain insights and

collaborate with fellow data management professionals

3


4

Peter Aiken, PhD• 25+ years of experience in data

management• Multiple international awards &

recognition• Founder, Data Blueprint (datablueprint.com)

• Associate Professor of IS, VCU (vcu.edu)

• President, DAMA International (dama.org)

• 8 books and dozens of articles• Experienced w/ 500+ data

management practices in 20 countries• Multi-year immersions with

organizations as diverse as the US DoD, Nokia, Deutsche Bank, Wells Fargo, and the Commonwealth of Virginia

2

Data Systems Integration & Business Value Part 3: Warehousing

Presented by Peter Aiken, Ph.D.


1. Data management overview

2. Motivation for warehousing integration technologies (reporting->BI->Analytics)

3. What are warehousing integration technologies?

4. Warehousing and architecture focus

5. The use of meta models

6. Guiding principles & best practices

7. Take aways, references and Q&A

Tweeting now: #dataed

Data Systems Integration & BV Part 3: Warehousing

6


















7

Data Program Coordination

Feedback

DataDevelopment


StandardData

Five Integrated DM Practice AreasOrganizational Strategies

Goals

BusinessData

Business Value

Application Models & Designs

Implementation

Direction

Guidance

8

OrganizationalData Integration

DataStewardship

Data SupportOperations

Data Asset Use

IntegratedModels

Leverage data in organizational activities

Data management processes andinfrastructure

Combining multipleassets to produceextra value

Organizational-entity subject area dataintegration

Provide reliable data access

Achieve sharing of data within a business area


Five Integrated DM Practice AreasManage data coherently.

Share data across boundaries.

Assign responsibilities for data.Engineer data delivery systems.

Maintain data availability.

Data Program Coordination

Organizational Data Integration

Data Stewardship Data Development

Data Support Operations

9


Hierarchy of Data Management Practices (after Maslow)

• 5 Data management practices areas / data management basics ...

• ... are necessary but insufficient prerequisites to organizational data leveraging applications that is self actualizing data or advanced data practices Basic Data Management Practices

– Data Program Management– Organizational Data Integration– Data Stewardship– Data Development– Data Support Operations

http://3.bp.blogspot.com/-ptl-9mAieuQ/T-idBt1YFmI/AAAAAAAABgw/Ib-nVkMmMEQ/s1600/maslows_hierarchy_of_needs.png

Advanced Data

Practices• MDM• Mining• Big Data• Analytics• Warehousing• SOA

Warehousing

• Data Management Body of Knowledge (DMBOK)– Published by DAMA International, the professional

association for Data Managers (40 chapters worldwide)

– Organized around primary data management functions focused around data delivery to the organization and several environmental elements

• Certified Data Management Professional (CDMP)– Series of 3 exams by DAMA International and

ICCP– Membership in a distinct group of

fellow professionals– Recognition for specialized knowledge in a

choice of 17 specialty areas– For more information, please visit:

• www.dama.org, www.iccp.org


DAMA DM BoK & CDMP

11


Series Context• Certain systems are more data

focused than others. Usually their primary focus is on accomplishing integration of disparate data. In these cases, failure is most often attributable to the adoption of a single technological pillar (silver bullet). The three webinars in the Data Systems Integration and Business Value series are designed to illustrate that good systems development more often depends on at least three DM disciplines (pie wedges) in order to provide a solid foundation.

• Data Systems Integration & Business Value – Pt. 1: Metadata Practices– Pt. 2: Cloud-based Integration– Pt. 3: Warehousing, et al.

12

Uses


Part 1: Metadata Take Aways• Metadata unlocks the value of data, and therefore

requires management attention [Gartner 2011]

• Metadata is the language of data governance• Metadata defines the essence of integration challenges

SourcesMetadata Governance

Metadata Engineering

Metadata Delivery

Metadata Practices

MetadataStorage

13

Specialized Team Skills


Part 2: Take Aways• Data governance, architecture,

quality, development maturity are necessary but insufficient prerequisites to successful data cloud implementation

• A variety of cloud options will influence cloud and data architectures in general– You must understand your architecture

and strategy in order to evaluate the options

• Data must be reengineered to be – Less– Better quality– More shareable – for the cloud

• Failure to do these will result in more business value for the cloud vendors/service providers and less for your organization


Summary: Data Warehousing & Business Intelligence Management

15











16











17


• Bank accounts are of varying value and risk

• Cube by – Social status– Geographical location– Net value, etc.

• Balance return on the loan with risk of default

18

• How to evaluate the portfolio as a whole?– Least risk loan may be to the very wealthy, but there are a very

limited number – Many poor customers, but greater risk

• Solution may combine types of analyses– When to lend, interest rate charged

Example: Portfolio Analysis


Target Isn't Just Predicting Pregnancies

19

http://rmportal.performedia.com/node/1373


15 years ago, CarMax started as a way to make the car buying experience simple, fair, and fun. Today CarMax is a FORTUNE 500 retailer and one of FORTUNE’s “100 Best Companies to Work For.” And we are hiring talented individuals who are interested in:--solving original, wide-ranging, and open-ended business problems--not only discovering new insights, but successfully implementing them--making a significant mark on a growing company--developing the fundamental skills for a rewarding business career

If that sounds like you, the Strategy Analyst position is the unique opportunity you’ve been looking for. The strategy team at CarMax currently consists of over 40 analysts, many of whom are recent college graduates from top schools with a variety of academic backgrounds (computer science, economics, English, engineering, journalism, math, political science). These analysts lead advances and decisions in several key business areas:-Inventory and pricing—what is the optimal selection of inventory, how do we acquire it, what should we pay for it, what should we price it for?-Expansion planning—which markets should we enter and how do we store those markets? Will each $10-30 million store investment generate a sufficient economic return?-Credit strategy—how can our bank (CarMax Auto Finance) approve more customers for loans and convert more approvals to sales?-Marketing and consumer insight—how do we reach our customers, increase traffic to our stores, and best use the internet to drive sales and build our brand-Industry and competitive research—what middle- and long-term risks are we exposed to, and how best do we prepare to respond? -Production—how do we increase vehicle reconditioning quality while reducing cost and production time?-Sales process and workforce—what is the best way to serve customers in our stores, and how do we manage, motivate and compensate our sales team?

Even early in your career at CarMax, you will have the responsibility to own an area of the business and will be expected to improve it. For example, one undergraduate recruit used data analysis to reformulate our retail pricing strategy, pitched and sold his idea to the senior executive team, and implemented a new system nationwide in his first 6 months with the company. That is the kind of impact you can make at CarMax. And as you do this, you will work closely with the senior executives and analytical managers to develop the fundamental and advanced skills that underpin a successful career in business. In fact, most of our managers in the strategy group started at CarMax as analysts, and our VP of Strategy and Analysis started his career here through our undergraduate recruiting program. While an MBA is not required to advance or contribute at CarMax, analysts who have chosen to pursue a business degree have enjoyed superior acceptance rates at their first choice schools, including Harvard, Chicago, UVa, Columbia, and Duke.

Your opportunities to develop, contribute, and lead as an analyst at CarMax are as great as the company’s opportunity to grow. While CarMax is already the largest used car retailer in the country (with over $8 billion in sales and over 90 superstores across the country), we have only 2% of the 1 to 6-year-old used car market, which, at $280 billion annually, is bigger than the home improvement or consumer electronics industries. CarMax is already growing at 15% a year, and over the next 10 years plans to have 250-300 stores and achieve $25+ billion in annual sales. As an analyst, you can be an integral part of that growth, all while enjoying a casual and friendly environment, a diverse group of talented associates, a healthy work-life balance, and excellent compensation and benefits.

An ideal candidate will have--Demonstrated top caliber analytic and problem solving skills --History of achievement demonstrated by top 15% GPA, with a quantitative major(s), and/or other recognition such as scholarships, awards, honor societies -- Passion for business and desire to develop into a strong business leader

We encourage you to apply. For more information, please visit us at the career fair, on our website (www.carmax.com/collegerecruiting), or email us at [email protected]://www.seas.virginia.edu/careerdevelopment/index.php?option=com_careerfairstudent&task=detailView&employerId=216&eventId=3

- datablueprint.com

CarMax Example Job Posting

24

own an area of the business and will be expected to improve it

--solving original, wide-ranging, and open-ended business problems--not only discovering new insights, but successfully implementing them--making a significant mark on a growing company--developing the fundamental skills for a rewarding business career


DW, Analytics, BI, Meta-Integration TechnologiesDefinitions• Beyond the nuts and bolts of data

management• Analysis of information that had

not been integrated previouslyBusiness Intelligence• Dates at least to 1958• Support better business decision

making• Technologies, applications and

practices for the collection, integration, analysis, and presentation of business information

• Also described as decision support

21

From The DAMA Guide to the Data Management Body of Knowledge © 2009 by DAMA International

Data Warehousing• Operational extract, cleansing,

transformation, load, and associated control processes for integrating disparate data into a single conceptual database


22

Definitions, cont’d• Study of data to discover and

understand historical patterns to improve future performance

• Use of mathematics in business

• Analytics closely resembles statistical analysis and data mining

– based on modeling involving extensive computation.

• Some fields within the area of analytics are

– enterprise decision management, marketing analytics, predictive science, strategy science, credit risk analysis and fraud analytics.


23

from MicroStrategy, Better Business Decisions Every Day: Integrating Business Reporting & Analysis

Example: Set Analysis


Polling Question #1

Do you have start data warehouse, data marts and/or other warehousing forms of integration?

a) Last year (2012)b) This year (2013)c) Next Year (2014)d) Nope

24











25











26


• Inmon:–"A subject oriented, integrated, time variant, and non-

volatile collection of summary and detailed historical data used to support the strategic decision-making processes of the organization."

• Kimball:–"A copy of transaction data specifically structured for

query and analysis."• Key concepts focus on:

–Subjects–Transactions–Non-volatility–Restructuring

Warehousing Definitions

27


Top 10 Data Warehouse Failure Causes1. The project is over budget2. Slipped schedule3. Functions and

capabilities not implemented

4. Unhappy users5. Unacceptable performance6. Poor availability7. Inability to expand 8. Poor quality data/reports9. Too complicated for users10.Project not cost justified

28

from The Data Administration Newsletter, www.tdan.com


29

Basic Data Warehouse Analysis

• Emphasis on the cube

• Permits different users to "slice and dice" subsets of data

• Viewing from different perspectives



30

Warehouse Analysis

• Users can "drill" anywhere

• Entire collection is accessible

• Summaries to transaction-level detail



Oracle

31

from The DAMA Guide to the Data Management Body of Knowledge © 2009 by DAMA International


R& D Applications(researcher supported, no documentation)

Finance Application(3rd GL, batch system, no source)

Payroll Application(3rd GL)

Payroll Data(database)

FinanceData

(indexed)

Personnel Data(database)

R & DData(raw)

Mfg. Data(home grown

database) Mfg. Applications(contractor supported)

Marketing Application(4rd GL, query facilities, no reporting, very large)

Marketing Data(external database)

Personnel App.(20 years old,

un-normalized data)

32

Multiple Sources of (for example) Customer Data


Corporate Information Factory Architecture

33




34



35




36




MetaMatrix Integration Example

37

• EII Enterprise Information Integration

– between ETL and EAI - delivers tailored views of information to users at the time that it is required


Linked Data

38

Linked Data is about using the Web to connect related data that wasn't previously linked, or using the Web to lower the barriers to linking data currently linked using other methods. More specifically, Wikipedia defines Linked Data as "a term used to describe a recommended best practice for exposing, sharing, and connecting pieces of data, information, and knowledge on the Semantic Web using URIs and RDF."

linkeddata.org











39











40


Kimball's DW Chess Pieces

41



3

Courtesy of: http://www.infosys.com/industries/healthcare/industryofferings/Pages/healthcare -data-warehousing.aspx

Data Warehousing


3

Descriptive Ask: What happened? What is happening? Find: Structured data Show: Profiles, Bar/Pie charts, Narrative

Predictive Ask: What will happen? Why will it happen? Find: Structured/unstructured data Show: Risk Profiles, Pros/Cons, Care Recs

Prescriptive Ask: What should I do? Why should I do it? Find: Unstructured/structured data Show: Strategic Goals, Support Recs

u Organization-wide

u Volume and Noise

u Utility

u Meaningful scoring

u Actionable recs

u Realistic goals

u Support

u Manage & measure

Analytics in Health Care


3

Descriptive Ask: What happened? What is happening? Find: Structured data Show: Profiles, Bar/pie charts, Narrative

Predictive Ask: What will happen? Why will it happen? Find: Structured/unstructured data Show: Risk Profiles, Pros/Cons, Care Recs

Prescriptive Ask: What should I do? Why should I do it? Find: Unstructured/structured data Show: Strategic Goals, Support Recs

BioMarin Licenses Factor VIII Gene Therapy Program for HemophiliaNovel Gene Therapy Approach to Hemophilia BSangamo BioSciences Receives $6.4 Million Strategic Partnership Award From California Institute for Regenerative Medicine to Develop ZFP Therapeutic®

Treating Hemophilia in the 2010s

Hemophilia Management


45

Styles of Business Intelligence



Health Care Provider Data Warehouse• 1.8 million members• 1.4 million providers• 800,000 providers no key• 2.2% prov_number = 9 digits (required)• 29% prov_ssn ≠ 9 digits• 1 User

46

"I can take a roomful of MBAs and accomplish this analysis faster!"


Top Causes of Data Warehouse Failure• Poor Quality Data

–Many more values of gender code than (M/F)

• Incorrectly Structured Data

–Providing the correct answer to the wrong question

• Bad Warehouse Design

–Overly complex

47



Indiana Jones: Raiders Of The Lost Ark

48


49

Business Intelligence Features

Problematic Data Quality


5 Key Business Intelligence Trends1. There's so much data, but too little

insight. More data translates to a greater need to manage it and make it actionable.

2. Market consolidation means fewer choices for business intelligence users.

3. Business Intelligence expands from the Board Room to the front lines. Increasingly, business intelligence tools will be available at all levels of the corporation

4. The convergence of structured and unstructured data Will create better business intelligence.

5. Applications will provide new views of business intelligence data. The next generation of business intelligence applications is moving beyond the pie charts and bar charts into more visual depictions of data and trends.

50

http://www.cio.com/article/150450/Five_Key_Business_Intelligence_Trends_You_Need_to_Know?page=2&taxonomyId=3002


Polling Question #2

Do you have?

a) A single enterprise data warehouse

b) Coordinated data marts

c) Bothd) Uncoordinated

effortse) None

51











52











53


Source:http://dmreview.com/article_sub.cfm?articleID=1000941 used with permission

Meta Data Models

54


Metadata Data ModelSCREENELEMENTscreen element id #data item id #screen element descr.

INTERFACEELEMENTinterface element id #data item id #interface element descr.

INPUTELEMENTinput element id #data item id #input element descr.

OUTPUTELEMENToutput element id #data item id #output element descr.

MODELVIEWmodel view element id #data item id #model view element des.

DEPENDENCYdependency elem id #data item id #process id #dependency description

CODEcode id #data item id #stored data item #code location

INFORMATIONinformation id #data item id #information descr.information request

PROCESSprocess id #data item id #process description

USER TYPEuser type id #data item id #information id #user type description

LOCATIONlocation id #information id #printout element id #process id #stored data items id #user type id #location description

PRINTOUTELEMENTprintout element id #data item id #printout element descr.

STORED DATA ITEMstored data item id #data item id #location id #stored data description

DATA ITEMdata item id #data item description

55


WarehouseProcess

WarehouseOpera-on

Transforma-on

XMLRecord-‐Oriented

Mul-DimensionalRela-onal

BusinessInforma-on

So?wareDeployment

ObjectModel(Core, Behavioral, Rela-onships, Instance)

WarehouseManagement

Resources

Analysis

Object-‐Oriented

(ObjectModel)

Foundation

OLAPData Mining

Informa-onVisualiza-on

BusinessNomenclature

DataTypes Expressions

KeysIndex

TypeMapping

Overview of CWM Metamodel

http://www.omg.org/technology/documents/modeling_spec_catalog.htm

56


Marco & Jennings's Complete Meta Data Model

Source:http://dmreview.com/article_sub.cfm?articleID=1000941 used with permission

57











58











59



Goals and Principles1. To support and enable

effective business analysis and decision making by knowledgeable workers

2. To build and maintain the environment/infrastructure to support business intelligence activities, specifically leveraging all the other data management functions to cost effectively deliver consistent integrated data for all BI activities

60


• Understand BI information needs

• Define and maintain the DW/BI architecture

• Process data for BI

• Implement data warehouse/data marts

• Implement BI tools and user interfaces

• Monitor and tune DW processes

• Monitor and tune BI activities and performance

61


Activities


Primary Deliverables • DW/BI Architecture

• Data warehouses, marts, cubes etc.

• Dashboards-scorecards

• Analytic applications

• Files extracts (for data mining, etc.)

• BI tools and user environments

• Data quality feedback mechanism/loop

62



Roles and ResponsibilitiesSuppliers:• Executives/managers• Subject Matter Experts• Data governance council• Information consumers• Data producers• Data architects/analysts


Participants:• Executives/managers• Data Stewards• Subject Matter Experts• Data Architects• Data Analysts• Application Architects• Data Governance Council• Data Providers• Other BI Professionals

63


Technology • ETL• Change Management Tools • Data Modeling Tools• Data Profiling Tools• Data Cleansing Tools• Data Integration Tools• Reference Data Management Applications• Master Data Management Applications• Process Modeling Tools• Meta-data Repositories• Business Process and Rule Engines


64


Guiding Principles1. Obtain executive commitment and

support. 2. Secure business SMEs. 3. Be business focused and driven. Let

the business drive the prioritization.4. Demonstrate data quality is essential.5. Provide incremental value.

65


6. Transparency and self service. 7. One size does not fit all: Find the right tools and products for each of

your segments.8. Think and architect globally, act and build locally.9. Collaborate with and integrate all other data initiatives, especially

those for data governance, data quality and metadata.10. Start with the end in mind. 11. Summarize and optimize last, not first.


6 Best Practices for Data Warehousing

66

1.Do some initial architecture envisioning.

2.Model the details just in time (JIT).

3.Prove the architecture early.

4.Focus on usage.

5.Organize your work by requirements.

6.Active stakeholder participation.

http://www.agiledata.org/essays/dataWarehousingBestPractices.html


Polling Question #3

Do you have a separate data warehouse department, sub-department, or group?

a) Yes b)No

67











68











69


Summary: Data Warehousing & Business Intelligence Management

70



Series Take Aways

71

• Metadata– Metadata unlocks the value of data, and therefore requires management

attention [Gartner 2011]– Metadata is the language of data governance– Metadata defines the essence of integration challenges

• Cloud– Data governance, architecture, quality, development maturity are necessary but

insufficient prerequisites to successful data cloud implementation– A variety of cloud options will influence cloud and data architectures in general– You must understand your architecture and strategy in order to evaluate the

options– Data must be reengineered to be: less; better quality; more shareable – Failure to do these will result in more business value for the cloud vendors/

service providers and less for your organization

• Warehousing– Business value must precede technical design


References

72


References

73


Additional References• http://www.information-management.com/infodirect/20050909/1036703-1.html • http://www.agiledata.org/essays/dataWarehousingBestPractices.html • http://www.cio.com/article/150450/

Five_Key_Business_Intelligence_Trends_You_Need_to_Know?page=2&taxonomyId=3002 • http://www.computerworld.com/s/article/9228736/

Business_Intelligence_and_analytics_Conquering_Big_Data?taxonomyId=9 • http://www.enterpriseirregulars.com/5706/the-top-10-trends-for-2010-in-analytics-business-

intelligence-and-performance-management/ • http://www.itbusinessedge.com/cm/blogs/vizard/taking-the-analytics-pressure-off-the-data-

warehouse/?cs=50698• http://www.informationweek.com/news/software/bi/240001922

74


Questions?

It’s your turn! Use the chat feature or Twitter (#dataed) to submit

your questions to Peter now.

75

+ =


Upcoming Events

76

October Webinar:SHOW ME THE MONEY: MONETIZING DATA MANAGEMENTOctober 8, 2013 @ 2:00 PM – 3:30 PM ET(11:00 AM-12:30 PM PT)

November Webinar:UNLOCK BUSINESS VALUE THROUGH REFERENCE & MDMNovember 12, 2013 @ 2:00 PM – 3:30 PM ET(11:00 AM-12:30 PM PT)

Sign up here:• www.datablueprint.com/webinar-schedule • www.Dataversity.net

Brought to you by: