Upload
maximilian-stephens
View
215
Download
1
Embed Size (px)
Citation preview
Semantic, Hierarchical, Online Clustering of Web Search Results
Yisheng Dong
2
Overview
Introduction Previous Related Works SHOC Approach Prototype System Conclusion
3
Motivation The Web is the biggest data source. Search engine is the most commonly used tool
for Web information retrieval. Its current status is far from the satisfaction.
Solution Clustering of Web search results would help a
lot. SHOC can generate both reasonable and
readable cluster.
Introduction
4
Basic requirements (clustering approach for web search result)
Semantic Each cluster should correspond to a concept. Avoid confining each Web page to only on
cluster. A label can describe the topic of cluster well.
Hierarchical Eye-browsing tree structure. Taking advantage of the relationship between
them. Online
Provide fresh clustering result “just-in-time”.
5
Previous Related Work
Scatter/Gather system traditional heuristic clustering algorithm. It has some limitations.
Based on hyperlink It needs to download and parse original Web
page. Cannot cluster immediately.
STC It is not appropriate for Oriental language. Extract many meaningless partial phrases. Synonymy and polysemy are not considered.
6
SOHC step
1. Data acquisition2. Data cleaning3. Feature extraction4. Identifying base clusters5. Combining base clusters
7
Data acquision
The data acquisition task here is actually meta-search.
Use 2-level parallelization mechanism
1. Call several engines simultaneously.2. Fetch all of its search result
simultaneously.
8
Data cleaning
Sentence boundaries are identified via the following. punctuation marks (e.g. ‘.’, ‘,’, ‘;’, ‘?’, etc.) HTML tags (e.g.<p>, <br>, <li>, <td> etc.)
Non-word tokens are stripped.(e.g. punctuation marks and HTML tags)
Redundant spaces are compressed. Stemming algorithm may be applied.
(for English text)
9
Feature extraction (Overview)
Words Most clustering algorithm treat a document as
“bag of words”. Ignoring word order and proximity.
Key phrases Advantage
Improve the quality of the clusters. Useful in constructing labels.
Data structures (key phrase discovery) Suffix tree
Related to the alphabet size of language. Suffix array
Scalable over alphabet size.
10
Feature extraction(key phrase discovery)
Completeness Left-completeness Right-completeness
Stability (Mutual Information) S =“c1c2∙∙∙cp”, SL =“c1∙∙∙cp-1”, SR =“c2∙∙∙cp”
Significance se(S) = freq(S) * g(|S|) g(x) 0 (x=1)
log2x (2≤x≤8) 3 (x>8)
freq(S) - )freq(S )freq(S
freq(S))(
RL SMI
11
Feature extraction (Suffix array)
Suffix array An array of all N suffixes, sorted alphabetically
LCP (Longest Common Prefix) Use to accelerate searching in text
<Suffix array and lcp of the “to_be_or_not_to_be”>
12
Feature extraction (Discover rcs)
void discover_rcs(){ typedef structure{ int ID; int frequency; } RCSTYPE; RSCTYPE rcs_stack[N]; // N is the document's
length Initialize rcs_stack; int sp = -1; // the stack pointer int i = 1; while(i < N+1) { if(sp < 0){ // the stack is empty if(lcp[i] > 0){ sp++; rcs_stack[sp].ID = i; rcs_stack[sp].frequency = 2; } i++; } else{ . . . } }}
int r = rcs_stack[sp].ID;if(lcp[r] < lcp[i]) { sp++; rcs_stack[sp].ID = i; rcs_stack[sp].frequency = 2; i++;}else if(lcp[r] == lcp[i]) { rcs_stack[sp].frequecny++; i++;}else{ Output rcs_stack[sp]; // ID & frequency int f = rcs_stack[sp].frequency; sp--; if(sp >= 0){ rcs_stack[sp].frequency = rcs_stack[sp].frequency + f -1; }}
13
Feature extraction (Intersect lcs_rcs)
void intersect_lcs_rcs(sorted lcs array, sorted rcs array)
{ int i =0, j=0; while(i<L && j < R) { string str_l = lcs[i].ID denoted LCS; string str_r = rcs[j].ID denoted RCS; if(str_l == str_r) { Output lcs[i]; i++; j++; } if(str_l < str_r){ i++; } if(str_l > str_r){ j++; } }}
rcs array
ID frequency RCS
1 2 _be
2 5 _
6 2 be
8 2 e
11 2 o_be
12 4 o
16 3 to_be
17 2 t
cs array
ID frequency CS
2 5 _
12 4 o
16 3 t
17 2 to_be
14
Identifying base clustersTerms
(key phrases) Documents
The association between terms and documents
15
Combining base clusters
Combine base cluster X and Y
if ( |X ∩ Y| / |X ∪ Y| > t1 ) { X and Y are merged into one cluster;}else { if ( |X| > |Y| ) { if ( |X ∩ Y| / |Y| > t2 ) { let Y become X’s child; } } else { if ( |X ∩ Y| / |X| > t2 ) { let X become Y’s child; } }}
Merging Label
if ( label x is a substring of label y ) { label_xy = label_y;}else if ( label_y is a substring of label_x ){ label_xy = label_x;}else{ label_xy = “ label_x + label_y ”;}
16
Prototype system
Crate a prototype system named WICE (Web Information Clustering Engine)
Doing well for dealing with the special problems related to Chinese
Output for query “object oriented” object oriented programming object oriented analysis, etc.
17
Conclusion
Main contribution The benefit of using key phrase. Method based on suffix array for key phrase. The concept of orthogonal clustering. The WICE system is designed and implemented.
Further works Detailed analysis. Further experimenting. Interpretation of experiment results. Comparing with other clustering algorithms.