Upload
bernard-hensley
View
218
Download
0
Tags:
Embed Size (px)
Citation preview
Crawling and Aligning Scholarly Presentations and Documents from the Web
By SARAVANAN.S09/09/2011
Under the guidance ofA/P Min-Yen Kan
04/20/23 1
Motivation• More articles more users• Searching for documents is difficult• Aim: Find pairs of presentations and documents automatically
04/20/23 2
System Architecture
Search Engine Wrapper
QueryQuery
“File type”( PDF, PPT or PS) operator is added with the user queryBefore sending it to Google.
Re-Ranking
Top results of GoogleOutput:(3-way)
1.Exact URL2.Message for
No-free files3.No result
Used Yee Fan’s Search Engine Wrapper – just Google subsystem
04/20/23 3
Methodology (1)
Re-Ranking•Computed similarity between user query and documents retrieved for re-ranking.•Methods used for computing similarity are Jaccard co-efficient, Bilingual Evaluation Understudy (BLEU).•Threshold value is used to restrict the system from considering low similarity scored documents.
Google’s Top
Results
Similarity Score
Computation
Re-RankingResults
Based on Similarity
ScoreSimilarity is computer between Query Title and each Google’s result Title, Snippet, URL.04/20/23 4
Jaccard Measure• Jaccard measure is used to compute similarity between Query
Title and Google’s result Title, Snippet, and URL.• Simple word by word matching.
Problems are:• Snippets have more words than title.• Union in Jaccard increases while intersection remains same.
Sentence1: Finding related pages in the world wide web.
Sentence2: Finding Related pages using the Link structure of the WWW.
04/20/23 5
BLEU metric
Why BLEU??• n-gram similarity of words.• Helps in accessing the sequential order of the words when
finding similarity between two sets.• Sequential order of words matters with snippetquery terms
may appear in a random position.
04/20/23 6
Rules
Special rules are used for better matching:
Rule1: Removing special symbols. (On/Off)Rule2: Stop-words removal (On/Off)Rule3: URL filter by .edu (On/Off)Rule4: Stemming (Porter stemming algorithm) (On/Off)
All these rules are used with both the methodologies.
04/20/23 7
MethodologyMIME-types:• To differentiate free PDF from subscription type, I used the MIME-types. It
returns the content-type of the URL.
Dataset collection:Queries from,• Computer science.• Medical science.• Architecture.• Mathematics.
04/20/23 8
Experiment• Experiments on
– Jaccard Measure.(All special rules are tested with On/Off).– BLEU measure (All special rules are tested with On/Off).– Query set with about 50 queries.– Threshold is set from 0.1 to 1.0 range for all experiments.– Highest recall with high threshold is considered.
• Experiment results– Jaccard similarity.– BLEU similarity.
04/20/23 9
Experiment result of JaccardRule Google
TargetThreshold Precision Recall F-Score
1 2 3 4
On Off Off Off Title 0.7 0.9818 0.9152 0.9473
Off On Off Off Title 0.6 0.7857 0.3728 0.5057
Off Off On Off Title 0.5 0.9272 0.8644 0.8947
Off Off Off On Title 0.5 0.9107 0.8644 0.8869
On On Off Off Title 0.6 0.7857 0.3728 0.5057
Off On On Off Title 0.5 0.6641 0.3728 0.5057
Off Off On On Title 0.4 0.9107 0.8644 0.8869
On Off Off On Title 0.5 0.9107 0.8644 0.8869
On On On Off Title 0.5 0.7857 0.3728 0.5057
Off On On On Title 0.5 0.7667 0.3898 0.5168
On Off On On Title 0.4 0.9107 0.8644 0.8869
On On Off On Title 0.7 0.8214 0.3898 0.5287
On On On On Title 0.5 0.9310 0.4576 0.6136
Off Off Off Off Title 0.7 0.9259 0.8474 0.8849
On Off On Off Title 0.5 0.9272 0.8644 0.8947
Off On Off On Title 0.3 0.9107 0.8644 0.8869
BestF-score
achieved
04/20/23 10
Best F-score
achieved
Experiment result of BLEU
04/20/23 11
Rule Google Target
Threshold Precision Recall F-Score
1 2 3 4
1 On Off Off Off Title 0.5 0.9272 0.8644 0.8947
2 Off On Off Off Title 0.3 0.7667 0.3898 0.5168
3 Off Off On Off Title 0.6 0.9412 0.8136 0.8727
4 Off Off Off On Title 0.5 0.9130 0.7118 0.8
5 On On Off Off Title 0.8 0.8148 0.3729 0.5116
6 Off On On Off Title 0.5 0.7586 0.3729 0.5
7 Off Off On On Title 0.5 0.9130 0.7119 0.8
8 On Off Off On Title 0.8 0.9074 0.8305 0.8672
9 On On On Off Title 0.8 0.8148 0.3728 0.5116
10 Off On On On Title 0.2 0.7586 0.3729 0.5
11 On Off On On Title 0.7 0.9074 0.8305 0.8672
12 On On Off On Title 0.8 0.8148 0.3728 0.5116
13 On On On On Title 0.6 0.8076 0.3559 0.4941
14 Off Off Off Off Title 0.5 0.9090 0.8478 0.8772
15 On Off On Off Title 0.5 0.9107 0.8644 0.8899
16 Off On Off On Title 0.5 0.76 0.3220 0.4523
Related WorkBase Reference:
– SlideSeer: a digital library of aligned document and presentation pairs, [Kan, JCDL’07].
– Learning to Rank for Information Retrieval. [Liu et al., WWW’09].– Kairos: Proactive Harvesting of Research Paper Metadata from
Scientific Conference Web Sites.[Hänse, ICADL’09]Approaches to Similarity Computation
– BLEU: a Method for Automatic Evaluation of Machine Translation. [Papineni et al., ACL July’02].
– BLEU algorithm for evaluation machine translations implementation.[Payson et al.]
04/20/23 12
Conclusion• Matching documents based on similarity score• Jaccard measure
-- Jaccard similarity computed over Query title and Document title with rule special symbol removed retrieves best articles.
-- Threshold:0.7-- F-score:0.9473
• BLEU metric-- BLEU similarity computed over Query title and Document
title with rule special symbol removed retrieves best articles.-- Threshold 0.5-- F-score:0.8947
04/20/23 13
Thank you
Comments are welcome
04/20/23 14