45
1 CPS 196.03: Information Management and Mining Association Rules and Frequent Itemsets

CPS 196.03: Information Management and Mining

  • Upload
    ling

  • View
    19

  • Download
    0

Embed Size (px)

DESCRIPTION

CPS 196.03: Information Management and Mining. Association Rules and Frequent Itemsets. Let Us Begin with an Example. A common marketing problem: examine what people buy together to discover patterns. What pairs of items are unusually often found together at Kroger checkout? - PowerPoint PPT Presentation

Citation preview

Page 1: CPS 196.03: Information Management and Mining

1

CPS 196.03: Information Management and Mining

Association Rules and Frequent Itemsets

Page 2: CPS 196.03: Information Management and Mining

2

Let Us Begin with an Example

A common marketing problem: examine what people buy together to discover patterns.

1. What pairs of items are unusually often found together at Kroger checkout?• Answer: diapers and beer.

2. What books are likely to be bought by the same Amazon customer?

Page 3: CPS 196.03: Information Management and Mining

3

Caveat A big risk when data mining is that

you will “discover” patterns that are meaningless.

Statisticians call it Bonferroni’s principle: (roughly) if you look in more places for interesting patterns than your amount of data will support, you are bound to find “false patterns”.

Page 4: CPS 196.03: Information Management and Mining

4

Rhine Paradox --- (1) David Rhine was a parapsychologist in

the 1950’s who hypothesized that some people had Extra-Sensory Perception.

He devised an experiment where subjects were asked to guess 10 hidden cards --- red or blue.

He discovered that almost 1 in 1000 had ESP --- they were able to get all 10 right!

Page 5: CPS 196.03: Information Management and Mining

5

Rhine Paradox --- (2) He told these people they had ESP

and called them in for another test of the same type.

Alas, he discovered that almost all of them had lost their ESP.

What did he conclude? Answer on next slide.

Page 6: CPS 196.03: Information Management and Mining

6

Rhine Paradox --- (3) He concluded that you shouldn’t

tell people they have ESP; it causes them to lose it.

Page 7: CPS 196.03: Information Management and Mining

7

“Association Rules”

Market BasketsFrequent ItemsetsA-priori Algorithm

Page 8: CPS 196.03: Information Management and Mining

8

The Market-Basket Model A large set of items, e.g., things

sold in a supermarket. A large set of baskets, each of

which is a small set of the items, e.g., the things one customer buys on one day.

Page 9: CPS 196.03: Information Management and Mining

9

Association Rule Mining

tran1 cust33 p2, p5, p8tran2 cust45 p5, p8, p11tran3 cust12 p1, p9tran4 cust40 p5, p8, p11tran5 cust12 p2, p9tran6 cust12 p9

transaction

id customer

id products

bought

salesrecords:

• Trend: Products p5, p8 often bought together

market-basketdata

Page 10: CPS 196.03: Information Management and Mining

10

Support Simplest question: find sets of items

that appear “frequently” in the baskets.

Support for itemset I = the number of baskets containing all items in I.

Given a support threshold s, sets of items that appear in > s baskets are called frequent itemsets.

Page 11: CPS 196.03: Information Management and Mining

11

Example Items={milk, coke, pepsi, beer, juice}. Support = 3 baskets.

B1 = {m, c, b} B2 = {m, p, j}B3 = {m, b} B4 = {c, j}B5 = {m, p, b} B6 = {m, c, b, j}B7 = {c, b, j} B8 = {b, c}

What are the possible itemsets? The Lattice of itemsets

How would you find the frequent itemsets?

Page 12: CPS 196.03: Information Management and Mining

12

Example Frequent itemsets: {m}, {c}, {b},

{j}, {m, b}, {c, b}, {j, c}.

Page 13: CPS 196.03: Information Management and Mining

13

Applications --- (1) Real market baskets: chain stores

keep terabytes of information about what customers buy together. Tells how typical customers navigate

stores, lets them position tempting items. Suggests tie-in “tricks,” e.g., run sale on

diapers and raise the price of beer. High support needed, or no $$’s .

Page 14: CPS 196.03: Information Management and Mining

14

Applications --- (2) “Baskets” = documents; “items” =

words in those documents. Lets us find words that appear together

unusually frequently, i.e., linked concepts. “Baskets” = sentences, “items” =

documents containing those sentences. Items that appear together too often could

represent plagiarism.

Page 15: CPS 196.03: Information Management and Mining

15

Applications --- (3) “Baskets” = Web pages; “items” =

linked pages. Pairs of pages with many common

references may be about the same topic. Ex: think of our two data mining textbooks

“Baskets” = Web pages p ; “items” = pages that link to p . Pages with many of the same links may be

mirrors or about the same topic. Ex: think of people with similar interests

Page 16: CPS 196.03: Information Management and Mining

16

Important Point “Market Baskets” is an abstraction

that models any many-many relationship between two concepts: “items” and “baskets.” Items need not be “contained” in baskets.

The only difference is that we count co-occurrences of items related to a basket, not vice-versa.

Page 17: CPS 196.03: Information Management and Mining

17

Scale of Problem WalMart sells 100,000 items and

can store billions of baskets. The Web has over 100,000,000

words and billions of pages.

Page 18: CPS 196.03: Information Management and Mining

18

Association Rules If-then rules about the contents of

baskets. {i1, i2,…,ik} → j means: “if a basket

contains all of i1,…,ik then it is likely to contain j.

Confidence of this association rule is the probability of j given i1,…,ik.

Page 19: CPS 196.03: Information Management and Mining

19

ExampleB1 = {m, c, b} B2 = {m, p, j}B3 = {m, b} B4 = {c, j}B5 = {m, p, b} B6 = {m, c, b, j}B7 = {c, b, j} B8 = {b, c}

An association rule: {m, b} → c. Confidence = 2/4 = 50%.

+__ +

Page 20: CPS 196.03: Information Management and Mining

20

Interest The interest of an association rule

is the absolute value of the amount by which the confidence differs from what you would expect, were items selected independently of one another.

Page 21: CPS 196.03: Information Management and Mining

21

ExampleB1 = {m, c, b} B2 = {m, p, j}B3 = {m, b} B4 = {c, j}B5 = {m, p, b}B6 = {m, c, b, j}B7 = {c, b, j} B8 = {b, c}

For association rule {m, b} → c, item c appears in 5/8 of the baskets.

Interest = | 2/4 - 5/8 | = 1/8 --- not very interesting.

Page 22: CPS 196.03: Information Management and Mining

22

Relationships Among Measures Rules with high support and

confidence may be useful even if they are not “interesting.” We don’t care if buying bread causes

people to buy milk, or whether simply a lot of people buy both bread and milk.

But high interest suggests a cause that might be worth investigating.

Page 23: CPS 196.03: Information Management and Mining

23

Finding Association Rules A typical question: “find all association

rules with support ≥ s and confidence ≥ c.” Note: “support” of an association rule is the

support of the set of items it mentions. Hard part: finding the high-support

(frequent ) itemsets. Checking the confidence of association rules

involving those sets is relatively easy.

Page 24: CPS 196.03: Information Management and Mining

24

Finding Association Rules Two-step approach:

1. Frequent Itemset Generation– Generate all itemsets whose support minsup

2. Rule Generation– Generate high confidence rules from each

frequent itemset, where each rule is a binary partitioning of a frequent itemset

Frequent itemset generation is still computationally expensive

Page 25: CPS 196.03: Information Management and Mining

25

Computation Model Typically, data is kept in a “flat

file” rather than a database system. Stored on disk. Stored basket-by-basket. Expand baskets into pairs, triples, etc.

as you read baskets.

Page 26: CPS 196.03: Information Management and Mining

26

Computation Model --- (2) The true cost of mining disk-resident

data is usually the number of disk I/O’s.

In practice, association-rule algorithms read the data in passes --- all baskets read in turn.

Thus, we measure the cost by the number of passes an algorithm takes.

Page 27: CPS 196.03: Information Management and Mining

27

Main-Memory Bottleneck In many algorithms to find frequent

itemsets we need to worry about how main memory is used. As we read baskets, we need to count

something, e.g., occurrences of pairs. The number of different things we can

count is limited by main memory. Swapping counts in/out is a disaster.

Page 28: CPS 196.03: Information Management and Mining

28

Finding Frequent Pairs The hardest problem often turns

out to be finding the frequent pairs.

We’ll concentrate on how to do that, then discuss extensions to finding frequent triples, etc.

Page 29: CPS 196.03: Information Management and Mining

29

The Lattice of ItemSetsnull

AB AC AD AE BC BD BE CD CE DE

A B C D E

ABC ABD ABE ACD ACE ADE BCD BCE BDE CDE

ABCD ABCE ABDE ACDE BCDE

ABCDE

Given d items, there are 2d possible candidate itemsets

Page 30: CPS 196.03: Information Management and Mining

30

Naïve Algorithm A simple way to find frequent pairs

is: Read file once, counting in main

memory the occurrences of each pair.• Expand each basket of n items into its

n (n -1)/2 pairs. Fails if #items-squared exceeds

main memory.

Page 31: CPS 196.03: Information Management and Mining

31

Details of Main-Memory Counting

There are two basic approaches:1. Count all item pairs, using a triangular

matrix.2. Keep a table of triples [i, j, c] = the

count of the pair of items {i,j } is c. (1) requires only (say) 4 bytes/pair;

(2) requires 12 bytes, but only for those pairs with >0 counts.

Page 32: CPS 196.03: Information Management and Mining

32

4 per pair

Method (1) Method (2)

12 peroccurring pair

Page 33: CPS 196.03: Information Management and Mining

33

Details of Approach (1) Number items 1,2,… Keep pairs in the order {1,2},

{1,3},…, {1,n }, {2,3}, {2,4},…,{2,n }, {3,4},…, {3,n },…{n -1,n }.

Find pair {i, j } at the position (i –1)(n –i /2) + j – i.

Total number of pairs n (n –1)/2; total bytes about 2n 2.

Page 34: CPS 196.03: Information Management and Mining

34

Details of Approach (2) You need a hash table, with i and j as

the key, to locate (i, j, c) triples efficiently. Typically, the cost of the hash structure can

be neglected. Total bytes used is about 12p, where p is

the number of pairs that actually occur. Beats triangular matrix if at most 1/3 of

possible pairs actually occur.

Page 35: CPS 196.03: Information Management and Mining

35

A-Priori Algorithm --- (1) A two-pass approach called a-priori

limits the need for main memory. Key idea: monotonicity : if a set of

items appears at least s times, so does every subset. Contrapositive for pairs: if item i does

not appear in s baskets, then no pair including i can appear in s baskets.

Page 36: CPS 196.03: Information Management and Mining

36

Found to be Infrequent

null

AB AC AD AE BC BD BE CD CE DE

A B C D E

ABC ABD ABE ACD ACE ADE BCD BCE BDE CDE

ABCD ABCE ABDE ACDE BCDE

ABCDE

Illustrating Apriori Principlenull

AB AC AD AE BC BD BE CD CE DE

A B C D E

ABC ABD ABE ACD ACE ADE BCD BCE BDE CDE

ABCD ABCE ABDE ACDE BCDE

ABCDEPruned supersets

Page 37: CPS 196.03: Information Management and Mining

37

Consider the following market-basket data

Market-Basket transactions

TID Items

1 Bread, Milk

2 Bread, Diaper, Beer, Eggs

3 Milk, Diaper, Beer, Coke 4 Bread, Milk, Diaper, Beer

5 Bread, Milk, Diaper, Coke

Illustrating Apriori Principle

Page 38: CPS 196.03: Information Management and Mining

38

Illustrating Apriori Principle

Item CountBread 4Coke 2Milk 4Beer 3Diaper 4Eggs 1

Itemset Count{Bread,Milk} 3{Bread,Beer} 2{Bread,Diaper} 3{Milk,Beer} 2{Milk,Diaper} 3{Beer,Diaper} 3

Itemset Count {Bread,Milk,Diaper} 3

Items (1-itemsets)

Pairs (2-itemsets)

(No need to generatecandidates involving Cokeor Eggs)

Triplets (3-itemsets)Minimum Support = 3

If every subset is considered, 6C1 + 6C2 + 6C3 = 41

With support-based pruning,6 + 6 + 1 = 13

Page 39: CPS 196.03: Information Management and Mining

39

A-Priori Algorithm --- (2) Pass 1: Read baskets and count in main

memory the occurrences of each item. Requires only memory proportional to #items.

Pass 2: Read baskets again and count in main memory only those pairs both of which were found in Pass 1 to be frequent. Requires memory proportional to square of

frequent items only.

Page 40: CPS 196.03: Information Management and Mining

40

Picture of A-Priori

Item counts

Pass 1 Pass 2

Frequent items

Counts ofcandidate pairs

Page 41: CPS 196.03: Information Management and Mining

41

Detail for A-Priori You can use the triangular matrix

method with n = number of frequent items. Saves space compared with storing

triples. Trick: number frequent items 1,2,

… and keep a table relating new numbers to original item numbers.

Page 42: CPS 196.03: Information Management and Mining

42

Frequent Triples, Etc. For each k, we construct two sets

of k –tuples: Ck = candidate k – tuples = those

that might be frequent sets (support > s ) based on information from the pass for k –1.

Lk = the set of truly frequent k –tuples.

Page 43: CPS 196.03: Information Management and Mining

43

C1 L1 C2 L2 C3Filter Filter ConstructConstruct

Firstpass

Secondpass

Page 44: CPS 196.03: Information Management and Mining

44

A-Priori for All Frequent Itemsets

One pass for each k. Needs room in main memory to

count each candidate k –tuple. For typical market-basket data and

reasonable support (e.g., 1%), k = 2 requires the most memory.

Page 45: CPS 196.03: Information Management and Mining

45

Frequent Itemsets --- (2) C1 = all items L1 = those counted on first pass to be

frequent. C2 = pairs, both chosen from L1. In general, Ck = k –tuples each k –1 of

which is in Lk-1. Lk = those candidates with support ≥ s.