Topics Related to Data Mining CS 4/59995. Information Retrieval Relevance Ranking Using Terms Relevance Using Hyperlinks Synonyms., Homonyms, and Ontologies

Topics Related to Data Mining

CS 4/59995

Information Retrieval

• Relevance Ranking Using Terms• Relevance Using Hyperlinks• Synonyms., Homonyms, and Ontologies• Indexing of Documents• Measuring Retrieval Effectiveness• Information Retrieval and Structured Data

Information Retrieval Systems• Information retrieval (IR) systems use a simpler data model

than database systems– Information organized as a collection of documents– Documents are unstructured, no schema

• Information retrieval locates relevant documents, on the basis of user input such as keywords or example documents– e.g., find documents containing the words “database

systems”• Can be used even on textual descriptions provided with non-

textual data such as images

Keyword Search• In full text retrieval, all the words in each document are

considered to be keywords. – We use the word term to refer to the words in a document

• Information-retrieval systems typically allow query expressions formed using keywords and the logical connectives and, or, and not– Ands are implicit, even if not explicitly specified

• Ranking of documents on the basis of estimated relevance to a query is critical– Relevance ranking is based on factors such as

• Term frequency– Frequency of occurrence of query keyword in document

• Inverse document frequency– How many documents the query keyword occurs in

» Fewer give more importance to keyword• Hyperlinks to documents

– More links to a document document is more important

Relevance Ranking Using Terms

• TF-IDF (Term frequency/Inverse Document frequency) ranking:– Let n(d) = number of terms in the document d– n(d, t) = number of occurrences of term t in the document d.

– Relevance of a document d to a term t

• The log factor is to avoid excessive weight to frequent terms

– Relevance of document to query Q

nn((dd))

nn((dd, , tt))1 +1 +TF TF ((dd, , tt) = ) = loglog

r r ((dd, , QQ) =) = TF TF ((dd, , tt))nn((tt))ttQQ

IDF=1/nIDF=1/n((tt), n(t) is the number of documents that contain the term t), n(t) is the number of documents that contain the term t

Relevance Ranking Using Terms (Cont.)• Most systems add to the above model

– Words that occur in title, author list, section headings, etc. are given greater importance

– Words whose first occurrence is late in the document are given lower importance

– Very common words such as “a”, “an”, “the”, “it” etc are eliminated• Called stop words

– Proximity: if keywords in query occur close together in the document, the document has higher importance than if they occur far apart

• Documents are returned in decreasing order of relevance score– Usually only top few documents are returned, not all

Synonyms and Homonyms

• Synonyms– E.g. document: “motorcycle repair”, query: “motorcycle maintenance”

• need to realize that “maintenance” and “repair” are synonyms

– System can extend query as “motorcycle and (repair or maintenance)”

• Homonyms– E.g. “object” has different meanings as noun/verb– Can disambiguate meanings (to some extent) from the context

• Extending queries automatically using synonyms can be problematic– Need to understand intended meaning in order to infer synonyms

• Or verify synonyms with user

– Synonyms may have other meanings as well

Indexing of Documents• An inverted index maps each keyword Ki to a set of documents Si

that contain the keyword– Documents identified by identifiers

• Inverted index may record – Keyword locations within document to allow proximity based ranking– Counts of number of occurrences of keyword to compute TF

• and operation: Finds documents that contain all of K1, K2, ..., Kn.– Intersection S1 S2 ..... Sn

• or operation: documents that contain at least one of K1, K2, …, Kn

– union, S1S2 ..... Sn,.

• Each Si is kept sorted to allow efficient intersection/union by merging

• “not” can also be efficiently implemented by merging of sorted lists

Word-Level Inverted File

lexicon posting

Measuring Retrieval Effectiveness• Information-retrieval systems save space by using index

structures that support only approximate retrieval. May result in:– false negative (false drop) - some relevant documents may

not be retrieved.– false positive - some irrelevant documents may be

retrieved.– For many applications a good index should not permit any

false drops, but may permit a few false positives.• Relevant performance metrics:

– precision - what percentage of the retrieved documents are relevant to the query.

– recall - what percentage of the documents relevant to the query were retrieved.

Measuring Retrieval Effectiveness (Cont.)

• Recall vs. precision tradeoff:• Can increase recall by retrieving many documents (down to a

low level of relevance ranking), but many irrelevant documents would be fetched, reducing precision

• Measures of retrieval effectiveness: – Recall as a function of number of documents fetched, or

– Precision as a function of recall – Equivalently, as a function of number of documents fetched– E.g. “precision of 75% at recall of 50%, and 60% at a recall of

75%”

• Problem: which documents are actually relevant, and which are not

Information Retrieval and Structured Data

• Information retrieval systems originally treated documents as a collection of words

• Information extraction systems infer structure from documents, e.g.:– Extraction of house attributes (size, address, number of

bedrooms, etc.) from a text advertisement– Extraction of topic and people named from a new article

• Relations or XML structures used to store extracted data– System seeks connections among data to answer

queries– Question answering systems

Probabilities and Statistic

Probabilities

)()()()( GEPGPEPGEP

)(1)( XPEP

1.

2.

Event E is defined as a any subset of

)()()()( GEPGPEPGEP

f(x) is called a probability distribution function (pdf)

Conditional Probabilities

)(

)()|(

GP

GEPGEP

)()()( GPEPGEP .

Expected value of X is

For continuous function f(x), the E(X) is

Conditional probability of E, provided that G occurred is

E and G are independent if and only if

Expected Value

E(X+Y) = E(X)+E(Y)

E(aX+b) = aE(X)+b

Variance• Var(X) = E(X-E(X))• It indicates how values of random variable are distributed around its expected value• Standard deviation of X is defined as • VAR(X+Y) = VAR(X) + VAR(Y)• VAR(aX+b) = VAR(X)b• P(|S - E(S)| r) VAR(S)/r (Chebyshev’s Ineequality)• Example:• {1,2,3,4,5,6}; p(i) =1/6 E(X)= 1*1/6+2*1/6+3*1/6+4*1/6+5*1/6+6*1/6• E(X)=1/6*21=3.5• (1,2,9,16,25,36) VAR(X) = E(X^2-2XE(X)+E^2(X)=E(X^2)-E^2(X)• E(X^2)=1/6(1+2+9+16+25+36)=1/6*89• E^2(X)=3.5*3.5=12.25• VAR(X)=89/6 -12.25=2.58• Standard deviation = sqrt(2.58)=1.61

2

2

)(XVAR2

2

Random Distributions

μ E(X) =

Var(X) = σ2

E(X) = npVar(X) = np(1-p)

Normal

Bernoulli

Normal Distributions

E(x) =

Random Distributions

P(X=x) = 1/(b-a)

Geometric

E(X) = 1/p; VAR(X) =(1-p)/p2

Poisson

E(X)=VAR(X)=m

Uniform

E(X)=(b-a)/2; VAR(X)= (b-a) /12 2

Correlation between age and mortality

Systolic Blood Pressure Distribution

Distribution of heart rate by systolic blood pressure

Data and their characteristics

Types of Attributes • There are different types of attributes

– Nominal• Examples: ID numbers, eye color, zip codes

– Ordinal• Examples: rankings (e.g., taste of potato chips on a scale

from 1-10), grades, height in {tall, medium, short}– Interval

• Examples: calendar dates, temperatures in Celsius or Fahrenheit.

– Ratio• Examples: temperature in Kelvin, length, time, counts

Properties of Attribute Values

• The type of an attribute depends on which of the following properties it possesses:– Distinctness: = – Order: < > – Addition: + - – Multiplication: * /

– Nominal attribute: distinctness– Ordinal attribute: distinctness & order– Interval attribute: distinctness, order & addition– Ratio attribute: all 4 properties

Attribute Type

Description Examples Operations

Nominal The values of a nominal attribute are just different names, i.e., nominal attributes provide only enough information to distinguish one object from another. (=, )

zip codes, employee ID numbers, eye color, sex: {male, female}

mode, entropy, contingency correlation, 2 test

Ordinal The values of an ordinal attribute provide enough information to order objects. (<, >)

hardness of minerals, {good, better, best}, grades, street numbers

median, percentiles, rank correlation, run tests, sign tests

Interval For interval attributes, the differences between values are meaningful, i.e., a unit of measurement exists. (+, - )

calendar dates, temperature in Celsius or Fahrenheit

mean, standard deviation, Pearson's correlation, t and F tests

Ratio For ratio variables, both differences and ratios are meaningful. (*, /)

temperature in Kelvin, monetary quantities, counts, age, mass, length, electrical current

geometric mean, harmonic mean, percent variation

Discrete and Continuous Attributes

• Discrete Attribute– Has only a finite or countably infinite set of values– Examples: zip codes, counts, or the set of words in a collection

of documents – Often represented as integer variables. – Note: binary attributes are a special case of discrete attributes

• Continuous Attribute– Has real numbers as attribute values– Examples: temperature, height, or weight. – Practically, real values can only be measured and represented

using a finite number of digits.– Continuous attributes are typically represented as floating-point

variables.

Data Matrix

• If data objects have the same fixed set of numeric attributes, then the data objects can be thought of as points in a multi-dimensional space, where each dimension represents a distinct attribute

• Such data set can be represented by an m by n matrix, where there are m rows, one for each object, and n columns, one for each attribute

1.12.216.226.2512.65

1.22.715.225.2710.23

Thickness LoadDistanceProjection of y load

Projection of x Load

1.12.216.226.2512.65

1.22.715.225.2710.23

Thickness LoadDistanceProjection of y load

Projection of x Load

Data Quality

• What kinds of data quality problems?• How can we detect problems with the data? • What can we do about these problems?

• Examples of data quality problems: – Noise and outliers – missing values – duplicate data

Noise

• Noise refers to modification of original values– Examples: distortion of a person’s voice when

talking on a poor phone and “snow” on television screen

Two Sine Waves Two Sine Waves + Noise

Outliers

• Outliers are data objects with characteristics that are considerably different than most of the other data objects in the data set

Data Preprocessing

• Aggregation

• Sampling

• Dimensionality Reduction

• Feature subset selection

• Feature creation

• Discretization and Binarization

• Attribute Transformation

Aggregation

• Combining two or more attributes (or objects) into a single attribute (or object)

• Purpose– Data reduction

• Reduce the number of attributes or objects– Change of scale

• Cities aggregated into regions, states, countries, etc– More “stable” data

• Aggregated data tends to have less variability

Sampling • Sampling is the main technique employed for data

selection.– It is often used for both the preliminary investigation of

the data and the final data analysis.

• Statisticians sample because obtaining the entire set of data of interest is too expensive or time consuming.

• Sampling is used in data mining because processing

the entire set of data of interest is too expensive or time consuming.

Sampling …

• The key principle for effective sampling is the following: – using a sample will work almost as well as

using the entire data sets, if the sample is representative

– A sample is representative if it has approximately the same property (of interest) as the original set of data

Types of Sampling• Simple Random Sampling

– There is an equal probability of selecting any particular item

• Sampling without replacement– As each item is selected, it is removed from the population

• Sampling with replacement– Objects are not removed from the population as they are selected for

the sample. • In sampling with replacement, the same object can be picked up

more than once

• Stratified sampling– Split the data into several partitions; then draw random samples from

each partition

Curse of Dimensionality

• When dimensionality increases, data becomes increasingly sparse in the space that it occupies

• Definitions of density and distance between points, which is critical for clustering and outlier detection, become less meaningful

• Randomly generate 500 points

• Compute difference between max and min distance between any pair of points

Discretization

• Three types of attributes:– Nominal — values from an unordered set– Ordinal — values from an ordered set– Continuous — real numbers

• Discretization: divide the range of a continuous attribute into

intervals– Some classification algorithms only accept

categorical attributes.– Reduce data size by discretization– Prepare for further analysis

Discretization and Concept hierarchy

• Discretization – reduce the number of values for a given continuous

attribute by dividing the range of the attribute into intervals. Interval labels can then be used to replace actual data values.

• Concept hierarchies – reduce the data by collecting and replacing low level

concepts (such as numeric values for the attribute age) by higher level concepts (such as young, middle-aged, or senior).

Discretization





Discretization and generation for numeric data

• Binning

• Histogram analysis

• Entropy-based discretization

• Segmentation by natural partitioning

Discretization

Sort Attribute

Select cut Point

Evaluate Measure

Satisfied

Split/Merge

NO

Yes

Stop

NO

DONE

Simple Discretization Methods: Binning

• Equal-width (distance) partitioning:– It divides the range into N intervals of equal size:

uniform grid– if A and B are the lowest and highest values of the

attribute, the width of intervals will be: W = (B-A)/N.– The most straightforward– But outliers may dominate presentation– Skewed data is not handled well.

• Equal-depth (frequency) partitioning:– It divides the range into N intervals, each containing

approximately same number of samples– Good data scaling– Managing categorical attributes can be tricky.

Discretization





Entropy-Based Discretization

• Given a set of samples S, if S is partitioned into two intervals S1 and S2 using boundary T, the entropy after partitioning is

• The boundary that minimizes the entropy function over all possible boundaries is selected as a binary discretization.

• The process is recursively applied to partitions obtained until some stopping criterion is met, e.g.,

• Experiments show that it may reduce data size and improve classification accuracy

E S TS

EntS

EntS S S S( , )| |

| |( )

| |

| |( ) 1

12

2

Ent S E T S( ) ( , )

Specification of a set of attributes

Concept hierarchy can be automatically generated based on the number of distinct values per attribute in the given attribute set. The attribute with the most distinct values is placed at the lowest level of the hierarchy.

country

province_or_ state

city

street

15 distinct values

65 distinct values

3567 distinct values

674,339 distinct values

Similarity and Dissimilarity

• Similarity

– Numerical measure of how alike two data objects are.

– Is higher when objects are more alike.

– Often falls in the range [0,1]

• Dissimilarity

– Numerical measure of how different are two data objects

– Lower when objects are more alike

– Minimum dissimilarity is often 0

– Upper limit varies

• Proximity refers to a similarity or dissimilarity

Similarity/Dissimilarity for Simple Attributes

p and q are the attribute values for two data objects.

Euclidean Distance

• Euclidean Distance

Where n is the number of dimensions (attributes) and pk and qk are, respectively, the kth attributes (components) or data objects p and q.

n

kkk qpdist

1

2)(

Euclidean Distance

0

1

2

3

0 1 2 3 4 5 6

p1

p2

p3 p4

point x yp1 0 2p2 2 0p3 3 1p4 5 1

Distance Matrix

p1 p2 p3 p4p1 0 2.828 3.162 5.099p2 2.828 0 1.414 3.162p3 3.162 1.414 0 2p4 5.099 3.162 2 0

Minkowski Distance

Minkowski Distance is a generalization of Euclidean Distance

Where r is a parameter, n is the number of dimensions (attributes) and pk and qk are, respectively, the kth attributes (components) or data objects p and q.

rn

k

rkk qpdist

1

1)||(

Minkowski Distance

Distance Matrix

point x yp1 0 2p2 2 0p3 3 1p4 5 1

L1 p1 p2 p3 p4p1 0 4 4 6p2 4 0 2 4p3 4 2 0 2p4 6 4 2 0

L2 p1 p2 p3 p4p1 0 2.828 3.162 5.099p2 2.828 0 1.414 3.162p3 3.162 1.414 0 2p4 5.099 3.162 2 0

Covariance

• Covariance=E((x-E(x)(y –E(y))

• Describes a some sort of dependency between variables.

• Describes how the X and Y change together

Common Properties of a Distance

• Distances, such as the Euclidean distance, have some well known properties.

1. d(p, q) 0 for all p and q and d(p, q) = 0 only if p = q. (Positive definiteness)

2. d(p, q) = d(q, p) for all p and q. (Symmetry)3. d(p, r) d(p, q) + d(q, r) for all points p, q, and r.

(Triangle Inequality)where d(p, q) is the distance (dissimilarity) between points (data objects), p and q.

• A distance that satisfies these properties is called a metric

Common Properties of a Similarity

• Similarities, also have some well known properties.

1. s(p, q) = 1 (or maximum similarity) only if p = q.

2. s(p, q) = s(q, p) for all p and q. (Symmetry)

where s(p, q) is the similarity between points (data objects), p and q.

Similarity Between Binary Vectors

• Common situation is that objects, p and q, have only binary attributes

• Compute similarities using the following quantitiesM01 = the number of attributes where p was 0 and q was 1M10 = the number of attributes where p was 1 and q was 0M00 = the number of attributes where p was 0 and q was 0M11 = the number of attributes where p was 1 and q was 1

• Simple Matching and Jaccard Coefficients SMC = number of matches / number of attributes

= (M11 + M00) / (M01 + M10 + M11 + M00)

J = number of 11 matches / number of not-both-zero attributes values = (M11) / (M01 + M10 + M11)

SMC versus Jaccard: Example

p = 1 0 0 0 0 0 0 0 0 0 q = 0 0 0 0 0 0 1 0 0 1

M01 = 2 (the number of attributes where p was 0 and q was 1)




SMC = (M11 + M00)/(M01 + M10 + M11 + M00) = (0+7) / (2+1+0+7) = 0.7

J = (M11) / (M01 + M10 + M11) = 0 / (2 + 1 + 0) = 0

Data Warehousing and OLAP Technology for Data Mining

• What is a data warehouse?

• Data warehouse architecture

• Data warehouse implementation

• Further development of data cube technology

• From data warehousing to data mining

Data Warehousing

• Large organizations have complex internal organizations, and have data stored at different locations, on different operational (transaction processing) systems, under different schemas

• Data sources often store only current data, not historical data• Corporate decision making requires a unified view of all

organizational data, including historical data• A data warehouse is a repository (archive) of information gathered

from multiple sources, stored under a unified schema, at a single site– Greatly simplifies querying, permits study of historical trends– Shifts decision support query load away from transaction

processing systems

Data Warehouse vs. Operational DBMS

• OLTP (on-line transaction processing)

– Major task of traditional relational DBMS

– Day-to-day operations: purchasing, inventory, banking, manufacturing, payroll, registration, accounting, etc.

• OLAP (on-line analytical processing)

– Major task of data warehouse system

– Data analysis and decision making

• Distinct features (OLTP vs. OLAP):

– User and system orientation: customer vs. market

– Data contents: current, detailed vs. historical, consolidated

– Database design: ER + application vs. star + subject

– View: current, local vs. evolutionary, integrated

– Access patterns: update vs. read-only but complex queries

OLTP vs. OLAP OLTP OLAP

users clerk, IT professional knowledge worker

function day to day operations decision support

DB design application-oriented subject-oriented

data current, up-to-date detailed, flat relational isolated

historical, summarized, multidimensional integrated, consolidated

usage repetitive ad-hoc

access read/write index/hash on prim. key

lots of scans

unit of work short, simple transaction complex query

# records accessed tens millions

#users thousands hundreds

DB size 100MB-GB 100GB-TB

metric transaction throughput query throughput, response

Data Warehousing

Managing of a warehouse

• When and how to gather data– Source driven architecture: data sources

transmit new information to warehouse, either continuously or periodically (e.g. at night)

– Destination driven architecture: warehouse periodically requests new information from data sources

• What schema to use– Schema integration

Managing of a warehouse(Cont.)

• Data cleansing– E.g. correct mistakes in addresses

• E.g. misspellings, zip code errors– Merge address lists from different sources and purge duplicates

• Keep only one address record per household (“householding”)• How to propagate updates

– Warehouse schema may be a (materialized) view of schema from data sources

– Efficient techniques for update of materialized views• What data to summarize

– Raw data may be too large to store on-line– Aggregate values (totals/subtotals) often suffice– Queries on raw data can often be transformed by query optimizer

to use aggregate values

Conceptual Modeling of Data Warehouses

• Modeling data warehouses: dimensions & measures

– Star schema: A fact table in the middle connected to a set of

dimension tables

– Snowflake schema: A refinement of star schema where some

dimensional hierarchy to snowflake

– Fact constellations is normalized into a set of smaller dimension

tables, forming a shape similar : Multiple fact tables share

dimension tables, viewed as a collection of stars, therefore called

galaxy schema or fact constellation

Star Schema Example

Example of Snowflake Schema

time_keydayday_of_the_weekmonthquarteryear

time

location_keystreetcity_key

location

Sales Fact Table

time_key

item_key

branch_key

location_key

units_sold

dollars_sold

avg_sales

Measures

item_keyitem_namebrandtypesupplier_key

item

branch_keybranch_namebranch_type

branch

supplier_keysupplier_type

supplier

city_keycityprovince_or_streetcountry

city

Example of Fact Constellation

time_keydayday_of_the_weekmonthquarteryear

time

location_keystreetcityprovince_or_streetcountry

location

Sales Fact Table

time_key

item_key

branch_key

location_key

units_sold

dollars_sold

avg_sales

Measures

item_keyitem_namebrandtypesupplier_type

item

branch_keybranch_namebranch_type

branch

Shipping Fact Table

time_key

item_key

shipper_key

from_location

to_location

dollars_cost

units_shipped

shipper_keyshipper_namelocation_keyshipper_type

shipper

Online Analytical Processing• The operation of changing the dimensions used in a cross-tab is

called pivoting• Suppose an analyst wishes to see a cross-tab on item-name and

color for a fixed value of size, for example, large, instead of the sum across all sizes. – Such an operation is referred to as slicing.

• The operation is sometimes called dicing, particularly when values for multiple dimensions are fixed.

• The operation of moving from finer-granularity data to a coarser granularity is called a rollup.

• The opposite operation - that of moving from coarser-granularity data to finer-granularity data – is called a drill down.

Three-Dimensional Data Cube A data cube is a multidimensional generalization of a crosstab

Cannot view a three-dimensional object in its entirety but crosstabs can be used as views on a data cube

Cube: A Lattice of Cuboids

all

time item location supplier

time,item time,location

time,supplier

item,location

item,supplier

location,supplier

time,item,location

time,item,supplier

time,location,supplier

item,location,supplier

time, item, location, supplier

0-D(apex) cuboid

1-D cuboids

2-D cuboids

3-D cuboids

4-D(base) cuboid

Hierarchies on Dimensions Hierarchy on dimension attributes: lets dimensions to be viewed at different levels of detail E.g. the dimension DateTime can be used to aggregate by hour of

day, date, day of week, month, quarter or year

OLAP Implementation• The earliest OLAP systems used multidimensional arrays in

memory to store data cubes, and are referred to as multidimensional OLAP (MOLAP) systems.

• OLAP implementations using only relational database features are called relational OLAP (ROLAP) systems

• Hybrid systems, which store some summaries in memory and store the base data and other summaries in a relational database, are called hybrid OLAP (HOLAP) systems.

OLAP Implementation (Cont.)• Early OLAP systems precomputed all possible aggregates in order to

provide online response– Space and time requirements for doing so can be very high

• 2n combinations of group by– It suffices to precompute some aggregates, and compute others

on demand from one of the precomputed aggregates• Can compute aggregate on (item-name, color) from an

aggregate on (item-name, color, size) – For all but a few “non-decomposable” aggregates such as

median– is cheaper than computing it from scratch

• Several optimizations available for computing multiple aggregates– Can compute aggregate on (item-name, color) from an aggregate

on (item-name, color, size)

– Can compute aggregates on (item-name, color, size), (item-name, color) and (item-name) using a single sorting of the base data

Group By cube• The cube operation computes union of group by’s on every subset

of the specified attributes• E.g. consider the query

select item-name, color, size, sum(number)from salesgroup by cube(item-name, color, size)

This computes the union of eight different groupings of the sales relation:

{ (item-name, color, size), (item-name, color), (item-name, size), (color, size), (item-name), (color), (size), ( ) }

where ( ) denotes an empty group by list.• For each grouping, the result contains the null value

for attributes not present in the grouping.

Group BY Cube (con’t)

• The function grouping() can be applied on an attribute– Returns 1 if the value is a null value representing all, and returns

0 in all other cases.

select item-name, color, size, sum(number),grouping(item-name) as item-name-flag,grouping(color) as color-flag,grouping(size) as size-flag,

from salesgroup by cube(item-name, color, size)

• Can use the function decode() in the select clause to replace such nulls by a value such as all

– E.g. replace item-name in first query by decode( grouping(item-name), 1, ‘all’, item-name)

Alternative: A Data Mining Query Language - DMQL

• Cube Definition (Fact Table)define cube <cube_name> [<dimension_list>]:

<measure_list>• Dimension Definition ( Dimension Table )

define dimension <dimension_name> as (<attribute_or_subdimension_list>)

• Special Case (Shared Dimension Tables)– First time as “cube definition”– define dimension <dimension_name> as

<dimension_name_first_time> in cube <cube_name_first_time>

Defining a Star Schema in DMQL

define cube sales_star [time, item, branch, location]:dollars_sold = sum(sales_in_dollars), avg_sales =

avg(sales_in_dollars), units_sold = count(*)define dimension time as (time_key, day, day_of_week, month,

quarter, year)define dimension item as (item_key, item_name, brand, type,

supplier_type)define dimension branch as (branch_key, branch_name, branch_type)define dimension location as (location_key, street, city,

province_or_state, country)

Defining a Snowflake Schema in DMQL

define cube sales_snowflake [time, item, branch, location]:

dollars_sold = sum(sales_in_dollars), avg_sales = avg(sales_in_dollars), units_sold = count(*)

define dimension time as (time_key, day, day_of_week, month, quarter, year)

define dimension item as (item_key, item_name, brand, type, supplier(supplier_key, supplier_type))

define dimension branch as (branch_key, branch_name, branch_type)

define dimension location as (location_key, street, city(city_key, province_or_state, country))

Defining a Fact Constellation in DMQLdefine cube sales [time, item, branch, location]:

dollars_sold = sum(sales_in_dollars), avg_sales = avg(sales_in_dollars), units_sold = count(*)

define dimension time as (time_key, day, day_of_week, month, quarter, year)

define dimension item as (item_key, item_name, brand, type, supplier_type)

define dimension branch as (branch_key, branch_name, branch_type)define dimension location as (location_key, street, city,

province_or_state, country)define cube shipping [time, item, shipper, from_location, to_location]:

dollar_cost = sum(cost_in_dollars), unit_shipped = count(*)define dimension time as time in cube salesdefine dimension item as item in cube salesdefine dimension shipper as (shipper_key, shipper_name, location as

location in cube sales, shipper_type)define dimension from_location as location in cube salesdefine dimension to_location as location in cube sales

Measures: Three Categories

• distributive: if the result derived by applying the function to n aggregate values is the same as that derived by applying the function on all the data without partitioning.

• E.g., count(), sum(), min(), max().

• algebraic: if it can be computed by an algebraic function with M arguments (where M is a bounded integer), each of which is obtained

by applying a distributive aggregate function.• E.g., avg(), min_M(), standard_deviation().

• holistic: if there is no constant bound on the storage size needed to describe a subaggregate.

• E.g., median(), mode(), rank().

Data Warehouse Design Process

• Top-down, bottom-up approaches or a combination of both– Top-down: Starts with overall design and planning (mature)– Bottom-up: Starts with experiments and prototypes (rapid)

• From software engineering point of view– Waterfall: structured and systematic analysis at each step before

proceeding to the next– Spiral: rapid generation of increasingly functional systems, short

turn around time, quick turn around• Typical data warehouse design process

– Choose a business process to model, e.g., orders, invoices, etc.– Choose the grain (atomic level of data) of the business process– Choose the dimensions that will apply to each fact table record– Choose the measure that will populate each fact table record

Multi-Tiered ArchitectureMulti-Tiered Architecture

DataWarehouse

ExtractTransformLoadRefresh

OLAP Engine

AnalysisQueryReportsData mining

Monitor&

IntegratorMetadata

Data Sources Front-End Tools

Serve

Data Marts

Operational DBs

other

sources

Data Storage

OLAP Server

Efficient Processing OLAP Queries

• Determine which operations should be performed on the available cuboids:

– transform drill, roll, etc. into corresponding SQL and/or OLAP operations,

e.g, dice = selection + projection

• Determine to which materialized cuboid(s) the relevant operations should be

applied.

• Exploring indexing structures and compressed vs. dense array structures in

MOLAP

Metadata Repository• Meta data is the data defining warehouse objects. It has the following

kinds – Description of the structure of the warehouse

• schema, view, dimensions, hierarchies, derived data defn, data mart locations and contents

– Operational meta-data• data lineage (history of migrated data and transformation path),

currency of data (active, archived, or purged), monitoring information (warehouse usage statistics, error reports, audit trails)

– The algorithms used for summarization– The mapping from operational environment to the data warehouse– Data related to system performance

• warehouse schema, view and derived data definitions– Business data

• business terms and definitions, ownership of data, charging policies

Data Warehouse Back-End Tools and Utilities

• Data extraction:– get data from multiple, heterogeneous, and external sources

• Data cleaning:– detect errors in the data and rectify them when possible

• Data transformation:– convert data from legacy or host format to warehouse format

• Load:– sort, summarize, consolidate, compute views, check integrity, and

build indicies and partitions• Refresh

– propagate the updates from the data sources to the warehouse

Discovery-Driven Exploration of Data Cubes

• Hypothesis-driven: exploration by user, huge search space

• Discovery-driven (Sarawagi et al.’98)

– pre-compute measures indicating exceptions, guide user in the

data analysis, at all levels of aggregation

– Exception: significantly different from the value anticipated,

based on a statistical model

– Visual cues such as background color are used to reflect the

degree of exception of each cell

– Computation of exception indicator (modeling fitting and

computing SelfExp, InExp, and PathExp values) can be

overlapped with cube construction

Examples: Discovery-Driven Data Cubes

Data Warehouse Usage

• Three kinds of data warehouse applications

– Information processing

• supports querying, basic statistical analysis, and reporting using crosstabs, tables, charts and graphs

– Analytical processing

• multidimensional analysis of data warehouse data

• supports basic OLAP operations, slice-dice, drilling, pivoting

– Data mining

• knowledge discovery from hidden patterns

• supports associations, constructing analytical models, performing classification and prediction, and presenting the mining results using visualization tools.

• Differences among the three tasks

From On-Line Analytical Processing to On Line

Analytical Mining (OLAM)

• Why online analytical mining?– High quality of data in data warehouses

• DW contains integrated, consistent, cleaned data– Available information processing structure surrounding data

warehouses• ODBC, OLEDB, Web accessing, service facilities, reporting

and OLAP tools– OLAP-based exploratory data analysis

• mining with drilling, dicing, pivoting, etc.– On-line selection of data mining functions

• integration and swapping of multiple mining functions, algorithms, and tasks.

• Architecture of OLAM

An OLAM Architecture

Data Warehouse

Meta Data

MDDB

OLAMEngine

OLAPEngine

User GUI API

Data Cube API

Database API

Data cleaning

Data integration

Layer3

OLAP/OLAM

Layer2

MDDB

Layer1

Data Repository

Layer4

User Interface

Filtering&Integration Filtering

Databases

Mining query Mining result