Normalizationand

Data MiningR&G Chapter 19

Lecture 27

Science is the knowledge of consequences, and dependence of one fact upon another.

Thomas Hobbes (1588-1679)

Administrivia

• Homework Due a week from Today– RubyOnRails help session Wed, 5-7pm, 310

Soda– (Thanks to Darren Lo & HKN)

• Final exam 3 weeks from tomorrow

Review: Functional Dependencies– Properties of the real world– Decide when to decompose relations– Help us find keys– Help us evaluate Design Tradeoffs

• Want to reduce redundancy, avoid anomalies• Want reasonable efficiency• Must avoid lossy decompositions

– F+: closure, all dependencies that can be inferred from a set F

– A+: attribute closure, all attributes functionally determined by the set of attributes A

– G: minimal cover, smallest set of FDs such that G+ == F+

Review: Normal Forms• A property of a single relation• Tells us something about redundancy in reln

• Reln R with FDs F is in BCNF if, for all X A in F+

A X (called a trivial FD), orX is a superkey for R.

• Reln R with FDs F is in 3NF if, for all X A in F+

A X (called a trivial FD), orX is a superkey of R, orA is part of some candidate key (not superkey!) for R.

(sometimes stated as “A is prime”)

Review: Decomposition

• If reln violates normal form, decompose– but must have lossless decomposition

• Lossless decomposition: – decomposition of R into X and Y is lossless if and only if

X Y is a key for either X or Y– If W Z holds over R and (W Z) is empty, then

decomposition of R into R-Z and WZ is loss-less.

• Algorithm: – For each FD W Z in R that violates normal form,

decompose R into R-Z and WZ. Repeat as needed.– Order not important, but can produce very different

results

Review: Dependency Preservation– decompose too much, and it might be necessary to join

tables to check FDs

– decomposition of R into X and Y is dependency preserving if (FX FY ) + = F +

• FX is all FDs involving only attributes in X• FY is all FDs involving only attributes in Y

– Not always obvious• ABC, A B, B C, C A, decomposed into AB and BC.• Is this dependency preserving? Is C A preserved?

– note: F + contains F {A C, B A, C B}, so…• FAB contains A B and B A; FBC contains B C and C B • So, (FAB FBC)+ contains C A

ExerciseConsider a database about Students:(StudentID, SS#, Name, Street Addr, City, State, Zip)abbreviated as: (D,S,N,R,C,T,Z), where D and S are

keysD DSNRCTZ, S DSNRCTZ, RCT Z, Z CT

• Is DSNRCTZ in BCNF? If not, decompose it until it is. Is the final decomposion dependency-preserving?

• Is DSNRCTZ in 3NF, If not, decompose it until it is. Is the final decomposion dependency-preserving?

ExerciseConsider a database about Students:(StudentID, SS#, Name, Street Addr, City, State, Zip)abbreviated as: (D,S,N,R,C,T,Z), where D and S are

keysD DSNRCTZ, S DSNRCTZ, RCT Z, Z CT

• Is DSNRCTZ in BCNF? If not, decompose it until it is. Is the final decomposion dependency-preserving?– no, RCT Z, RCT not key, decom to: DSNRCT &

RCTZ. – still no, Z CT, Z not key, decom to: DSNRCT,

ZCT & RZ, which is BCNF– but, join required to test RCT Z

ExerciseConsider a database about Students:(StudentID, SS#, Name, Street Addr, City, State, Zip)abbreviated as: (D,S,N,R,C,T,Z), where D and S are

keysD DSNRCTZ, S DSNRCTZ, RCT Z, Z CT

• Is DSNRCTZ in 3NF, If not, decompose it until it is. Is the final decomposion dependency-preserving?– no, RCT Z, RCT not key, Z not part of key

decom to: DSNRCT & RCTZ. – yes, 3NF, Z CT, CT part of key, (since RCT Z)– is D Z preserved? Yes, transitively, since D

RCT (1st relation), and RCT Z (2nd relation)

Minimal Cover for a Set of FDs• G: minimal cover, smallest set of FDs such that G+ == F+

– Closure of F = closure of G.– Right hand side of each FD in G is a single attribute.– If we modify G by deleting an FD or by deleting attributes from

an FD in G, the closure changes.

• Every FD in G is needed, and ``as small as possible’’ in order to get the same closure as F.

• e.g., F+ = {A B, B C, C A, B A, C B, A C}– several minimal covers: {A B, B A, C B, B C} (AB + BC)– or {A C, C A, B C, C B} (AC + BC)– or {A B, B A, C A, A C} (AB + AC)

• e.g., A B, ABCD E, EF GH, ACDF EG minimal cover:– A B, ACD E, EF G and EF H

BCNF and Dependency Preservation

• In general, there may not be a dependency preserving decomposition into BCNF.

• But, you can always find dependency-preserving decomposition into 3NF– Top down:

• Decompose until it is in 3NF• Compute minimal cover for FDs• If minimal cover contains a FD X Y is not preserved, add reln XY

– Bottom up:• Compute minimal cover• For each FD X Y in minimal cover, create reln XY

– Why does this work? Minimal cover doesn’t include redundant transitive dependencies, which don’t need to be preserved

Summary of FDs and Normalization

• FDs are properties of the real world– FDs tell us if a relation is in a Normal Form

• Normal forms tell us if there is any redundancy– but zero redundancy may mean inefficiency

• BCNF: each field contains information that cannot be inferred using only FDs. – ensuring BCNF is a good heuristic.

• Not in BCNF? Try decomposing into BCNF relations.– Must consider whether all FDs are preserved!

• Lossless-join, dependency preserving decomposition into BCNF impossible? Consider 3NF.

• Decompositions should be carried out while keeping performance requirements in mind.

• Note: even more restrictive Normal Forms exist (we don’t cover them in this course, but some are in the book.)

New Topic: Data Mining

Definition

Data mining is the exploration and analysis of large quantities of data in order to discover valid, novel, potentially useful, and ultimately understandable patterns in data.

Example pattern (Census Bureau Data):If (relationship = husband), then (gender = male). 99.6%

Definition (Cont.)

Data mining is the exploration and analysis of large quantities of data in order to discover valid, novel, potentially useful, and ultimately understandable patterns in data.

Valid: The patterns hold in general.

Novel: We did not know the pattern beforehand.

Useful: We can devise actions from the patterns.

Understandable: We can interpret and comprehend the patterns.

Why Use Data Mining?

Human analysis skills are inadequate:– Volume and dimensionality of the data– High data growth rate

Availability of:– Data– Storage– Computational power– Off-the-shelf software– Expertise

An Abundance of Data• Supermarket scanners, POS data• Preferred customer cards• Credit card transactions• Direct mail response• Call center records• ATM machines• Demographic data• Sensor networks• Cameras• Web server logs• Customer web site trails

More Computational Power• Moore’s Law:

In 1965, Intel Corporation cofounder Gordon Moore predicted that the density of transistors in an integrated circuit would double every year.(Later changed to reflect 18 months progress.)

• Experts on ants estimate that there are 1016 to 1017 ants on earth. In the year 1997, we produced one transistor per ant.

Much Commercial Support

• Many data mining tools– http://www.kdnuggets.com/software– http://www.purpleinsight.com

• Database systems with data mining support

• Visualization tools• Data mining process support• Consultants

Why Use Data Mining Today?Competitive pressure!“The secret of success is to know something that nobody

else knows.”Aristotle Onassis

• Competition on service, not only on price (Banks, phone companies, hotel chains, rental car companies)

• Personalization, CRM• The real-time enterprise• “Systemic listening”• Security, homeland defense

The Knowledge Discovery Process

Steps: Identify business problem Data mining Action Evaluation and measurement Deployment and integration into

businesses processes

Data Mining Step in Detail

2.1 Data preprocessing– Data selection: Identify target datasets

and relevant fields– Data cleaning

• Remove noise and outliers• Data transformation• Create common units• Generate new fields

2.2 Data mining model construction2.3 Model evaluation

Models can describe existing dataMake predictions about new data

Preprocessing and Mining

Original Data

TargetData

PreprocessedData

PatternsKnowledge

DataIntegration

and Selection

Preprocessing

ModelConstruction

Interpretation

Examples

• Insurance: which claims are likely to be fraud?

• Banks: which customers are likely to repay loans?

• Stores: which products do people buy together?

Data Mining Techniques• Supervised learning

– Classification and regression, describe correlative factors, predict values for new data

• Unsupervised learning– Clustering– Dependency modeling

• Associations, summarization, causality– Outlier and deviation detection– Trend analysis and change detection

• Visual Data Mining– Present the information in a visual form, offload the

analysis onto the human perceptual system

Supervised learning

• Need training data set with known outcome– e.g. here is a set of loans that were not repaid, and

other loans that were repaid

• Model is generated from the training set, tested on a separate test data set to determine accuracy

• Model can predict outcomes on new data, – can also explain predictive factors

• Examples include Decision Trees, Regression Trees, Naïve Baysian networks

Unsupervised Learning

• Give data to the algorithm, it does the rest

• Output might include clustered data, association rules, etc.

E.g. Agglomerative ClusteringAlgorithm:• Put each item in its own cluster (all singletons)• Find all pairwise distances between clusters• Merge the two closest clusters• Repeat until everything is in one cluster

Observations:• Results in a hierarchical clustering• Yields a clustering for each possible number of clusters• Greedy clustering: Result is not “optimal” for any

cluster size

Agglomerative Clustering Example

Density-Based Clustering • A cluster is defined as a connected dense

component.• Density is defined in terms of number of

neighbors of a point.• We can find clusters of arbitrary shape

Demo

Conclusions

• Data mining very useful for understanding large data sets

• Several approaches– Supervised– Unsupervised

• Can describe patterns, make predictions

• Many commercial packages

• Many free algorithms