A Combined Approach for Classification of Web Results based on Ranking & Clustering

Presented by:

Apeksha Khabia

Guided by:

Dr. M. B. Chandak

Contents

Introduction Page rank, weighted page rank Document clustering Algorithm for clustering and ranking Conclusion Future scope

Introduction

The web is most precious place for Information retrieval and Knowledge Discovery

Retrieving information through queries from a search engine is tedious

Solution is Web Mining web content mining web structure mining web uses mining

How to Generate Web Results?

Page Rank (PR)

Order the search results such that important documents move up and less important move down in the list

If a page has some important incoming links then its outgoing link also becomes important

Page Rank (PR) Rank score of a page p is evenly divided among

outgoing links

Modified PR in view Random Surfer Model – not all the users follow direct Links on WWW

Example of Page Rank (PR)

PR(A)= (1-d)+d((PR(B)/2+PR(C)/2 )

PR(B)= (1-d)+d( PR(A)/1+PR(C)/2 )

PR(C)= (1-d)+d( PR(B)/2)

IF d=0.5

PR(A)=1.2, PR(B)=1.2, PR(C)=0.8

Table : Iterative method

of page rank

Weighted Page Rank (WPR)

Assign larger rank values to more important pages instead of evenly dividing among its outgoing links.

Outlink page gets value according to its popularity

Document Clustering

 Automatic document organization, topic extraction and fast information retrieval or filtering

Documents are grouped together based upon measure of similarity of content or of hyperlinked structure

Clustering divides the results of a search for "cell" into groups like "biology," "battery," and "prison."

Document Clustering

Examples : K-means, hierarchical Clustering may be based on content alone, or

both on contents and links or only on links Two ways to define content based similarity

between the documents Resemblance Containment

Limitations of Ranking Approach

They give emphasis to links of the resultant pages No algorithm exists to combine the link score and

content score of the page into a single score Existing approaches return millions of documents

in an ordered format Rank based approaches give equal emphasis to

inlinks as well as outlinks of pages

Introduction of Combined Approach This mechanism takes advantage of importance

of inlinks over outlinks With the use of this user can put search results

into hierarchy of query related clusters Also the documents in each cluster can be ranked

to represent them according to their relevancy Such organization enables the user to effectively

limit his search area

Algorithm for Clustering and Ranking

Algorithm: Steps

Step 1: Get the URLs of the pages Step 2: Provide a similarity value sim(q, p) to

each returned document Step 3: Use sim(q, p) to cluster the

documents Step 4: Provide a rank score WSR(p) to the

documents of each cluster:

Output

Clusters of web pages documents are formed based on the similarity

Also the documents in each cluster are ranked to represent them according to their relevancy

Similarity Calculation between Web Pages Similarity of the document with the query means: what query terms are present in the document? where they are present? how many times? Calculated using cosine between vector of query

terms and vector of documents

Rank Calculation of Web pages

WSR- Weight and Similarity based Rank Back-links contribute more towards the

importance of a page rather than forward links WSR gives more importance to the inlinks of a

page Importance of the backlink page v of a page u,

given by

Rank Calculation of Web pages

Redefined formula for rank is given by

Clustering of Web pages

The clustering is purely based on the similarity values of the pages with respect to the user query

The number of clusters is not predefined The maximum number of pages that can be

in a cluster should be decided

Clustering of Web pages

Lower and upper value of similarity is identified from range of similarity values

Complete page set is divided into number of sets according to the similarity values lying within the partitioned ranges.

Rank(p)= WSR(p) + sim(q, p)

Applications of Clustering and Ranking Readability assessment - automatically

determining the degree of readability of a text, either to find suitable materials for different age groups or reader types or as part of a larger text simplification system

Genre classification - automatically determining the genre of a text

Tools Available

Weka Rapid Miner KNIME Orange

Conclusion

Ranking and clustering gives a way to organize the search results in the form of clusters, the pages in each cluster are further ranked to provide the most relevant and important pages on the top of the cluster

User search space decreases and he can get required content in short time

Future Ideas

Different data mining tools (KNIME, Rapid Miner, Weka) can be used to analyze the result for classification

Search query results for classification can be incorporated from multiple search engines

References Parneet Kaur, Sawtantar Singh Khurmi and Gurpreet Singh Josan, " Analysis

for Classification of Similar Documents among Various Websites using Rapid Miner“. In the proceedings of IEEE International Conference on Issues and Challenges in Intelligent Computing Techniques (ICICT), 2014

Neelam Duhan and A.K. Shanna, "A Novel Approach for Organizing Web Search Results using Ranking and Clustering". In the Proceedings of International Journal of Computer Applications, vol. 5, No. 10, pp. 1-9, August 2010.

O. Zamir, O. Etzioni. “Web document clustering: A feasibility demonstration”. Proceedings of the 19th International ACM SIGIR Conference on Research and Development of Information Retrieval (SIGIR'98), 46-54,1998.

References

Miguel Gomes da Costa Júnior, Zhiguo Gong, “Web Structure Mining: An introduction”. Proceedings of the IEEE International Conference on Information Acquisition, 2005, China.

Taher H. Haveliwala, Aristides Gionis, Dan Klein, Piotr Indyk, “Evaluating strategies for similarity search on the Web”. WWW2002, May, 2002, Honolulu, Hawaii, USA.ACM 1-58113-449-5/02/0005.

Thank You!!