Datawarehousing material

8/18/2019 Datawarehousing material

1/89

Data WarehouseConcepts&

Architecture

1By Monstercourses.com


2/89

2

Topics

• Data warehousing &

Architecture

• Data Mart

• ETL

• OLTP

• DSS• Database Design

• Star Schema

• Snow!a"e Schema• #seu! inormation

• $it a!!s

•

Summary By Monstercourses.com


3/89

A producer wants toknow….

What is the TotalRevenue for theyear 2009?

What is the TotalRevenue for theyear 2009?

Who are my customers

and what productsare they uyin!?

Who are my customersand what products

are they uyin!?

Which customers are most likely to !oto the competition ?

Which customers

are most likely to !oto the competition ?

What impact willnew products"serviceshave on revenueand mar!ins?

W

hat impact willnew products"serviceshave on revenue

and mar!ins?

hat product prom#otions have the i!!est

pact on revenue?

hat product prom#

tions have the i!!estpact on revenue?

W

hat is the moste$ective distriutionchannel?

W

hat is the moste$ective distriutionchannel?


4/89

%ata& %ata everywhereyet ...

% cant in' the 'ata % nee' 'ata is scattere' o(er the networ"

many (ersions) subt!e 'ierences

% cant get the 'ata % nee'

nee' an e*+ert to get the 'ata

% cant un'erstan' the 'ata % oun' a(ai!ab!e 'ata +oor!y 'ocumente'

% cant use the 'ata % oun' resu!ts are une*+ecte'

'ata nee's to be transorme' rom

one orm to other


5/89

'cenario (

A)* +vt ,td is a company withranches at -umai& %elhi&

*hennai and )an!lore. The 'ales-ana!er wants uarterly salesreport. /ach ranch has a

separate operational system.

,By Monstercourses.com


6/89

'cenario ( A)* +vt ,td.

-umai

%elhi

*hennai

)an!lore

'ales-ana!er

'ales per item type per ranchfor 1rst uarter.

-By Monstercourses.com


7/89

'olution (A)* +vt ,td.

• E*tract sa!es inormation rom each

'atabase.

• Store the inormation in a common re+ositoryat a sing!e site.

By Monstercourses.com


8/89

'olution (A)* +vt ,td.

-umai

%elhi

*hennai

)an!lore

%ataWarehouse

'ales-ana!er

uery 3Analysis tools

Report

/By Monstercourses.com


9/89

'cenario 2

4ne 'top 'hoppin! 'uper -arket hashu!eoperational dataase. Whenever

/5ecutives wants some report the4,T+ system ecomes slow and dataentry operators have to wait for some

time.



10/89

'cenario 2 4ne 'top'hoppin!

4perational%ataase

%ata /ntry 4perator

%ata /ntry 4perator

-ana!ementWait

Report



11/89

'olution 2

• E*tract 'ata nee'e' or ana!ysis romo+erationa! 'atabase.

• Store it in warehouse.

• eresh warehouse at regu!ar inter(a! so thatit contains u+ to 'ate inormation or ana!ysis.

• 3arehouse wi!! contain 'ata with historica!+ers+ecti(e.



12/89

'olution 2

4perationaldataase

%ataWarehouse

/5tractdata

%ata /ntry4perator

%ata /ntry4perator

-ana!er

Report

Transaction



13/89

'cenario 6

*akes 3 *ookies is a small& newcompany. +resident of the company

wants his company should !row. 7eneeds information so that he canmake correct decisions.



14/89

'olution 6

• %m+ro(e the 5ua!ity o 'ata beore

!oa'ing it into the warehouse.

• Perorm 'ata c!eaning an'

transormation beore !oa'ing the 'ata.

• #se 5uery ana!ysis too!s to su++ort

a'hoc 5ueries.



15/89

'olution 6

uery and Analysistool

+resident

/5pansion

8mprovement

sales

time

%ataarehouse

1,By Monstercourses.com


16/89

1-

What the users are sayin!...

• Data shou!' be integrate'

across the enter+rise

• Summary 'ata has a rea!

(a!ue to the organi7ation• 8istorica! 'ata ho!'s the "ey

to un'erstan'ing 'ata o(er

time• 3hat9i ca+abi!ities are

re5uire'



17/89

1

Application Areas

8ndustry Application

inance *redit *ard Analysis

8nsurance *laims& raud AnalysisTelecommunication

*all record Analysis

Transport ,o!istics -ana!ement

*onsumer !oods promotion Analysis

%ata 'erviceproviders

:alue added data

;tilities +ower usa!e Analysis



18/89

1/

Why 'eparate %ataWarehouse?

Performance – O+ 'bs 'esigne' & tune' or "nown t*s & wor"!oa's.

– :om+!e* OLAP 5ueries wou!' 'egra'e +er. or o+ t*s.

– S+ecia! 'ata organi7ation) access & im+!ementation metho'snee'e' or mu!ti'imensiona! (iews & 5ueries.

Function Missing 'ata; Decision su++ort re5uires historica! 'ata) which o+ 'bs 'o

not ty+ica!!y maintain.

Data conso!i'ation; Decision su++ort re5uires conso!i'ation


19/89

%ata Warehouse.. %e1ned

>A 'ata warehouse is a co!!ection o

cor+orate inormation) 'eri(e' 'irect!y

rom o+erationa! systems an' somee*terna! 'ata sources. %ts s+eciic

+ur+ose is to su++ort business

'ecisions) not business o+erations?



20/89


21/89

'u


22/89

Time :ariant

• Designate' Time $rame


23/89

8nte!ration

• %n terms o 'ata. – enco'ing structures.

– Measurement o

attributes.

– +hysica! attribute.

o 'ata

– naming con(entions.

– Data ty+e ormat

remark s



24/89

=on#:olatile

• *C!+D, Actions

"perationa' Syste

!ead

Insert

+pdate!ep'ace

Create

De'ete

• -o Data +pdate

Data Warehouse

Load!ead

!ead

!ead

!ead



25/89

*haracteristics of a %W

• Sub@ect9oriente' Data – co!!ects a!! 'ata or a sub@ect) rom 'ierent sources

• ea'9on!y e5uests

– !oa'e' 'uring o9hours) rea'9on!y 'uring 'ay hours

• %nteracti(e $eatures) a'9hoc 5uery

– !e*ib!e 'esign to han'!e s+ontaneous user 5ueries

• Pre9aggregate' 'ata

– to im+ro(e runtime +erormance• 8igh!y 'enorma!i7e' 'ata structures

– at tab!es with re'un'ant co!umns

2,By Monstercourses.com

%ata Warehousin!


26/89

%ata Warehousin!Architecture

/5tractTransform ,oadRefresh

'erve

/5ternal

'ources

4perational%s

Analysis

uery"Reportin!

%ata -inin!

-onitorin! 3

Administration

-etadataRepository

%ATA '4;R*/' T44,'

%ATA -ART'

4,A+ 'ervers

Reconcileddata

2-By Monstercourses.com


27/89

2

%W ,ayered Architecture



28/89

2/

What are 4perational'ystems?

• They are OLTP systems

• un mission critica!

a++!ications

• ee' to wor" withstringent +erormance

re5uirements or routine

tas"s

• #se' to run a businessF



29/89

20

4perational 'ystems

• un the business in rea! time

• Base' on u+9to9the9secon' 'ata

• O+timi7e' to han'!e !arge numberso sim+!e rea'Gwrite transactions

• O+timi7e' or ast res+onse to+re'eine' transactions

• #se' by +eo+!e who 'ea! withcustomers) +ro'ucts 99 c!er"s)

sa!es+eo+!e etc.• They are increasing!y use' by

customers



30/89

4

4,T+ vs %ata Warehouse

• OLTP – A++!ication Oriente'

– #se' to run business

– Detai!e' 'ata

– :urrent u+ to 'ate

– %so!ate' Data

– e+etiti(e access

– :!erica! #ser

• 3arehouse


31/89

41


• OLTP – Perormance Sensiti(e

– $ew ecor's accesse' at a

time


32/89

42


• OLTP – Transaction

through+ut is the+erormance metric

– Thousan's o users

• Data 3arehouse – Iuery through+ut is

the +erormance

metric

– 8un're's o users



33/89

44

To summari>e ...

• OLTP Systems are

use' to “run” a

business

• The Data 3arehouse

he!+s to “optimize”

the business



34/89

/T, …?

/5traction Transformation 3,oadin!



35/89

4,

Why /T,..%ata 8nte!rity +rolems

• Same +erson) 'ierent s+e!!ings

– Agarwa!) Agrawa!) Aggarwa! etc...

• Mu!ti+!e ways to 'enote com+any name

– Persistent Systems) PSPL) Persistent P(t. LTD.• #se o 'ierent names

– mumbai) bombay• Dierent account numbers generate' by 'ierent

a++!ications or the same customer

• e5uire' ie!'s !et b!an"• %n(a!i' +ro'uct co'es co!!ecte' at +oint o sa!e

– manua! entry !ea's to mista"es

– >in case o a +rob!em use 0000000?



36/89

8ntroduction

Source

System 1

Source

System 2

Source

System 4

Staging AreaData warehouse

E

T

L

E

T

L

E*traction) Transormation) a!i'ation) Loa'



37/89

/5traction

– Source Systems


38/89

Transformation

– #sage o too!s• eusabi!ity o Transormations

•eusabi!ity o Ma++ings

– Dierent too!s• %normatica

• 3arehouse Bui!'er

• ET%

• Sagent

• PLGSIL scri+ts

4/By Monstercourses.com


39/89

,oadin!

– Loa'ing $re5uency

– O+timi7e' Loa'ing• %n'e*ing

• Partitioning

– Aggregation• Sum

• A(erage• Ma*

– #+'ate Strategy

– Error 8an'!ing40By Monstercourses.com


40/89

6

%ata Transformation Terms

• Data :!eaning

• Data :on'itioning

• Data Scrubbing• Data Merging

• Data Aggregation



41/89

61

%ata Transformation Terms

• Data :!eaning

– %t is +rocess o the c!

– Sources or 'ata genera!!y in !egacy mainrames in

SAM) %MS) %DMS) DB2K more 'ata to'ay inre!ationa! 'atabases on #ni*

• :on'itioning

– The con(ersion o 'ata ty+es rom the source to the

target 'ata store


42/89

62

,oad Types

• Ongoing Data Loa' or %ncrementa!

Loa'ing

• Bu!" Loa'


43/89

64

%ata /5traction and*leansin!

• E*tract 'ata rom e*isting o+erationa!an' !egacy 'ata

• %ssues; – Sources o 'ata or the warehouse – Data 5ua!ity at the sources

– Merging 'ierent 'ata sources

– Data Transormation – 8ow to +ro+agate u+'ates


44/89

66

'cruin! %ata

• So+histicate' transormationtoo!s.

• #se' or c!eaning the 5ua!ity

o 'ata• :!ean 'ata is (ita! or the

success o the warehouse• E*am+!e

– Sesha'ri) Shesha'ri) Sesa'ri)Sesha'ri S.) Srini(asanSesha'ri) etc. are the same+erson



45/89

'TA8= AR/A # 'ome *larity

• Staging Area – o+tiona!

– to c!eanse the source 'ata – Acce+ts 'ata rom 'ierent sources

– Data mo'e! is re5uire' at staging area

–

Mu!ti+!e 'ata mo'e!s may be re5uire' or+ar"ing 'ierent sources an' or

transorme' 'ata to be +ushe' out to

warehouse



46/89

Types of %ata Warehouse

•

Enter+rise Data 3arehouse• Data Mart

Enterprise

Data Warehouse

Datamart Datamart Datamart



47/89

/nterprise data warehouse

• :ontains 'ata 'rawn rom mu!ti+!eo+erationa! systems

• Su++orts time9 series an' tren' ana!ysis

across 'ierent business areas• :an be use' as a transient storage area to

c!ean a!! 'ata an' ensure consistency

• :an be use' to +o+u!ate 'ata marts• :an be use' or e(ery'ay an' strategic

'ecision ma"ing

6/By Monstercourses.com


48/89

What is %ata -art?

• A 'ata mart is a subset o 'ata warehouse

that is 'esigne' or a +articu!ar !ine o

business) such as sa!es) mar"eting) or

inance.

• %n a 'e+en'ent 'ata mart) 'ata can be

'eri(e' rom an enter+rise9wi'e 'ata

warehouse. %n an in'e+en'ent 'ata mart)'ata can be co!!ecte' 'irect!y rom

sources.



49/89

%ata Warehouse vs. %ata-arts

3hat comes irst


50/89

%ata -art

•

Logica! subset o enter+rise 'atawarehouse

• Organi7e' aroun' a sing!e business

+rocess• Base' on granu!ar 'ata

• May or may not contain aggregates

•Ob@ect o ana!ytica! +rocessing by theen' user.

• Less e*+ensi(e an' much sma!!er than

a u!! b!own cor+orate 'ata warehouse.,1By Monstercourses.com

+hysical data warehouse


51/89

+hysical data warehouse%ata warehouse ##@ data marts

• S"+!C. DA(A

•

/5ternal• %ata

• 4perational %ata

• 'ta!in! Area

• %ata Warehouse • %ata -arts

• +hysical %ata Warehouse• %ata Warehouse ##@ %ata -arts

,2By Monstercourses.com

+hysical data warehouse


52/89

+hysical data warehouse%ata marts ##@ data warehouse

S"+!C. DA(A

/5ternal%ata

4perational %ata

'ta!in! Area

%ata Warehou

%ata -arts

+hysical %ata Warehouse0%ata -arts ##@ %ata Warehouse


+hysical %ata Warehouse


53/89

+hysical %ata Warehouse+arallel %ata Warehouse and

%ata -art

S"+!C. DA(A

/5ternal

%ata

4perational %ata

'ta!in! Area

%ata Wareho

%ata -arts

+hysical %ata Warehouse+arallel %ata Warehouse 3 %ata -arts


%W 8mplementation


54/89

%W 8mplementationApproaches

• To+ Down

• Bottom9u+

• :ombination o both

• Choices depend on:

– current inrastructure

– resources

– architecture

– O%

– %m+!ementation s+ee',,By Monstercourses.com


55/89

Top %own 8mplementation

,-By Monstercourses.com


56/89

)ottom ;p 8mplementation


%W 8 l i


57/89


To+ Down• More +!anning an'

'esign initia!!y• %n(o!(e +eo+!e rom

'ierent wor"9grou+s)'e+artments

• Data marts may be bui!t

!ater rom H!oba! D3• O(era!! 'ata mo'e! to

be 'eci'e' u+9ront

Bottom #+• :an +!an initia!!y without

waiting or g!oba!

inrastructure

• bui!t incrementa!!y

• can be bui!t beore or in

+ara!!e! with H!oba! D3

• Less com+!e*ity in

'esign

,/By Monstercourses.com

%W 8mplementation


58/89


To+ Down• :onsistent 'ata 'einition

an' enorcement obusiness ru!es across

enter+rise

• 8igh cost) !engthy

+rocess) time consuming

• 3or"s we!! when there is

centra!i7e' %S 'e+artment

res+onsib!e or a!! 8G3

an' resources

Bottom #+• Data re'un'ancy an'

inconsistency between'ata marts may occur

• %ntegration re5uires

great +!anning

• Less cost o 8G3 an'

other resources

• $aster +ay9bac"



59/89


Combined Approach• Determine 'egree o +!anning an' 'esign or

a g!oba! a++roach to integrate 'ata martsbeing bui!t by bottom9u+ a++roach

• De(e!o+ base !e(e! inrastructure 'einition or

g!oba! D3 at business !e(e!

• De(e!o+ +!an to han'!e 'ata e!ementsnee'e' by mu!ti+!e 'ata marts

• Bui!' a common 'ata store to be use' by

'ata marts an' g!oba! D3 -By Monstercourses.com


60/89

• Must i'entiy – Business +rocess to be su++orte'

– Hrain


61/89

*onventions used in%imensional modelin!

• $actsGMeasures


62/89

acts

• A act is a co!!ection o re!ate' 'ata

items) consisting o measures an'

conte*t 'ata.

• Each act ty+ica!!y re+resents a

business item) a business transaction)or an e(ent that can be use' in

ana!y7ing the business or business

+rocess.• $acts are measure') >continuous!y

(a!ue'?) ra+i'!y changing inormation.

:an be ca!cu!ate' an'Gor 'eri(e'.-4By Monstercourses.com


63/89

)asic concept of act Tale..

• The centra!i7e' tab!e in a star schema

is ca!!e' as $A:T tab!e. A act tab!e

ty+ica!!y has two ty+es o co!umns;those that contain acts an' those that

are oreign "eys to 'imension tab!es.

The +rimary "ey o a act tab!e isusua!!y a com+osite "ey that is ma'e u+

o a!! o its oreign "eys.

-6By Monstercourses.com


64/89

'o?....

• A tab!e that is use' to store business

inormation


65/89

%imensions

• A 'imension is a co!!ection o members

or units o the same ty+e o (iews.

• Dimensions 'etermine the conte*tua!

bac"groun' or the acts.

• Dimensions re+resent the way business

+eo+!e ta!" about the 'ata resu!ting

rom a business +rocess) e.g.) who)what) when) where) why) how

--By Monstercourses.com

%imension with respect to


66/89

%imension with respect toact

• Tab!e use' to store 5ua!itati(e 'ata

about act recor's

– 3ho – 3hat

– 3hen

–3here

– 3hy

-By Monstercourses.com

%imensions Tale


67/89

%imensions Tale

• Dimension tab!e is one that 'escribe the

business entities o an enter+rise)re+resente' as hierarchica!) categorica!

inormation such as time) 'e+artments)

!ocations) an' +ro'ucts.

-/By Monstercourses.com

'o? %imensions are


68/89

'o?... %imensions are

• :o!!ection o members or units o the

same ty+e o (iews.• 'etermine the conte*tua! bac"groun' or

the acts.

• the +arameters o(er which we want to+erorm OLAP


69/89

7ierarchies

A !ogica! structure that uses or'ere' !e(e!s as ameans o organi7ing 'ata. A hierarchy can be

use' to 'eine 'ata aggregationK or e*am+!e)

in a time 'imension) a hierarchy might be

use' to aggregate 'ata rom the Month !e(e!

to the Iuarter !e(e!) rom the Iuarter !e(e! to

the ear !e(e!.

A hierarchy can a!so be use' to 'eine ana(igationa! 'ri!! +ath) regar'!ess o whether

the !e(e!s in the hierarchy re+resent

aggregate' tota!s or not



70/89

7ierarchies

• A!!ow or the ro!!u+ o 'ata to more

summari7e' !e(e!s.

– Time• 'ay

• month

• 5uarter

• year


7i hi


71/89

7ierarchies


,e el


72/89

,evel

A +osition in a hierarchy. $or e*am+!e) a time'imension might ha(e a hierarchy that

re+resents 'ata at the Month) Iuarter) an'

ear !e(e!s.


-


73/89

-easures

•

A measure is a numeric attribute o aact) re+resenting the +erormance or

beha(iour o the business re!ati(e to

'imensions.

• The actua! numbers are ca!!e' as

(ariab!es.eg. sa!es in money) sa!es (o!ume) 5uantity su++!ie')

su++!y cost) transaction amount• A measure is 'etermine' by

combinations o the members o the

'imensions an' is !ocate' on acts.6By Monstercourses.com


74/89

• Star Schema is a re!ationa! 'atabase schema or

re+resenting mu!ti'imensiona! 'ata. %t is the sim+!est orm

o 'ata warehouse schema that contains one or more

'imensions an' act tab!es. %t is ca!!e' a star schema

because the entity9re!ationshi+ 'iagram between

'imensions an' act tab!es resemb!es a star where one act

tab!e is connecte' to mu!ti+!e 'imensions. The center o the

star schema consists o a !arge act tab!e an' it +oints

towar's the 'imension tab!es. The a'(antage o starschema is s!icing 'own) +erormance increase an' easy

un'erstan'ing o 'ata.

What is 'tar 'chema?


*ommon structures for


75/89

• Star – Sing!e act tab!e surroun'e' by 'enorma!i7e'

'imension tab!es

– The act tab!e +rimary "ey is the com+osite o

the oreign "eys


76/89

/5ample of 'tar 'chema


' l k ' h


77/89

• A snow!a"e schema is a term that 'escribes astar schema structure norma!i7e' through the

use o outrigger tab!es. i.e. 'imension tab!e

hierarchies are bro"en into sim+!er tab!es.

'now lake 'chema

/By Monstercourses.com

*ommon structures for% -


78/89

• Snow!a"e – Sing!e act tab!e surroun'e' by norma!i7e'

'imension tab!es

–orma!i7es 'imension tab!e to sa(e 'ata storages+ace.

– 3hen 'imensions become (ery (ery !arge

– Less intuiti(e) s!ower +erormance 'ue to @oins

• May want to use both a++roaches) es+ecia!!y

i su++orting mu!ti+!e en'9user too!s.

%ata -arts%enormali>e



79/89


80/89

'nowBake # %isadvanta!es

• orma!i7ation o 'imension ma"es it

'iicu!t or user to un'erstan'

• Decreases the 5uery +erormancebecause it in(o!(es more @oins

• Dimension tab!es are norma!!y sma!!er

than act tab!es 9 s+ace may not be ama@or issue to warrant snow!a"ing

/1By Monstercourses.com


81/89

Ceys …

• Primary Ceys – uni5ue!y i'entiy a recor'

• $oreign Ceys – +rimary "ey o another tab!e reerre' here

• Surrogate Ceys

– system9generate' "ey or 'imensions – "ey on its own has no meaning

– integer "ey) !ess s+ace


'chema 3 'now lake' h


82/89

'chema

In a star schea eery diension 0i'' hae a priary

ey2

In a star schea3 a diension tab'e 0i'' not hae any

parent tab'e2

Whereas in a sno0 4'ae schea3 a diension tab'e

0i'' hae one or ore parent tab'es2

5ierarchies 4or the diensions are stored in the

diensiona' tab'e itse'4 in star schea2

Whereas hierarchies are broen into separate tab'es in

sno0 4'ae schea2 (hese hierarchies he'ps to dri''

do0n the data 4ro topost hierarchies to the

'o0erost hierarchies2



83/89

)asic %imensional -odelin!


84/89

)asic %imensional -odelin!Techniues

• S!owing changing Dimensions

• :onirme' Dimensions

• Degenerate Dimensions

• un" Dimensions

/,By Monstercourses.com

'l l *h i %i i


85/89

'lowly *han!in! %imension

Dimensions that change o(er the +erio' o

time are ca!!e' S!ow!y :hanging

Dimensions. – $or instance) a +ro'uct +rice changes o(er timeK

– Peo+!e change their names or some reasonK

– :ountry an' State names may change o(er time.

S:D 9 Ty+es – Ty+e19O(erwirting the e*isting (a!ues

– Ty+e29Maintain the history o change' (a!ues

– Ty+e49Partia! history maintenance.

/-By Monstercourses.com


86/89

D k %i i


87/89

Dunk %imension

:reate s+ecia! 'imensions to ho!'

misce!!aneous attributes oun' in the source

'atabase

Scenario;Occasiona!!y) there are misce!!aneous attributes) such as

yesGno attributes or comment attributes) that 'ont it into

tight star schemas. ather than 'iscar'ing !ag ie!'s an'yesGno attributes) +!ace them in a @un" 'imension. %n

a''ition) you can han'!e comment an' o+en9en'e' te*t

attributes by creating a te*t9base' @un" 'imension

//By Monstercourses.com


88/89


89/89

Thank You

Documents

Datawarehousing material