Upload
ethel-beasley
View
225
Download
0
Embed Size (px)
DESCRIPTION
3 1.Introduction Clustering techniques are based on four concepts, data representation model, similarity measure, clustering model, and clustering algorithm Vector Space Document (VSD) model Suffix Tree Document (STD) model
Citation preview
1
Efficient Phrase-Based Document Similarity for Clustering
IEEE Transactions On Knowledge And Data Engineering, Vol. 20, No. 9, Page(s):1217-1229,2008 Speaker: Wei-Cheng WuData:2008/10/23
2
Outline1. Introduction2. The Phrase-Based Document
Similarity3. Experimental Results 4. Conclusions
3
1. Introduction
Clustering techniques are based on four concepts , data representation model , similarity measure , clustering model , and clustering algorithm
Vector Space Document (VSD) model Suffix Tree Document (STD) model
4
2. The Phrase-Based Document Similarity
Standard Suffix Tree Document Model and STC Algorithm
5
2. The Phrase-Based Document Similarity
EX:(1, 2,3) (1, 2) 2 0.5(1,2,3) (1, 2) 3
b c
b c
B BB B
6
The Phrase-Based Document Similarity Based on the STD Model
2. The Phrase-Based Document Similarity
7
Vector Space Document (VSD) model
2. The Phrase-Based Document Similarity
(1, ), (2, ),......., ( , )d w d w d w M d
(1)
( , ) (1 log ( , )) log(1 / ( ))w i d tf i d N df i
Example of Fig.1
( ) 3, ( ,1) 1df b tf b
( ,1) (1 log1) log(1 3/ 3) 0.693w b
8
2. The Phrase-Based Document Similarity
(2)
1 2 1 2, ,...., , , ,.....,x M y Md x x x d y y y
Let vectors
(3)
9
Properties of the STD Model
2. The Phrase-Based Document Similarity
10
2. The Phrase-Based Document Similarity
11
Property1. Each internal node of the suffix tree T represents an LCP of the document data set D, and each leaf node represents a suffix substring of a document in the data set D.
Property 2. Each first-level node in suffix tree T is labeled by a distinct phrase that appears at least once in the documents of data set D. The number of the first-level nodes is equal to the number of keywords (distinct single-word terms in the VSD model) in the data set D.
Property 3. Each phrase denoted by an internal node v at a higher level ( ) in suffix tree T contains at least two words. The length of the phrase (by words).
2. The Phrase-Based Document Similarity
vP2vL
2vP
12
3. Experimental Results
OHSUMED Document Collection RCV1 Document Collection 20-Newsgroups Document Collection Original STC algorithm GHAC (group-average HAC algorithm) with the phrase-based document similarity GHAC with the traditional single-word tf-idf cosine similarity K-NN clustering algorithm with the phrase-based document similarity
13
3. Experimental Results
14
1 2{ , ,..... }kC C C C is a clustering of data set D of N document
* * * *1 2{ , ,..... }lC C C C designate the “correct” class set of D
The recall of cluster j with respect to class i
* *( , ) /j i irec i j C C C
The precision of cluster j with respect to class i
*( , ) /j i jprec i j C C C
3. Experimental Results
15
3. Experimental Results
2 ( , ) ( , )( , )( , ) ( , )
prec i j rec i jF i jprec i j rec i j
*
1,..,1
max{ ( , )}l
i
i ki
CF F i j
N
1,..,1
max{ ( , )}k
j
i lj
CPurity prec i j
N
16
3. Experimental Results
17
3. Experimental Results
18
3. Experimental Results
19
The Performance Evaluation on Large Document Data Sets
we conducted a set of experiments on a large data set(DS8) that are generated from the RCV1 document collection .The data set DS8 contains 500 documents of category GSPO, M11, respectively, and all documents of other eight categories. The total number of documents is 4,759.
3. Experimental Results
20
3. Experimental Results
21
3. Experimental Results
22
3. Experimental Results
23
3. Experimental Results
24
4. Conclusions
The new phrase-based document similarity successfully connects the two document models and inherits their advantages.