Upload
layton-filmer
View
225
Download
6
Embed Size (px)
Citation preview
Collection Selection Overview
Coll1 Coll3 Coll2
document relevant to QueryA document relevant to QueryB document relevant to QueryC
UserX
QueryA
UserY UserZ
QueryB QueryCQueryA
Clones Broker Broker
BROKER ACTIONS
1) Receives query from user
2) Decides which collections are likely to contain resources relevant to the query, ranks them
3) Forwards query to top-n ranked collection servers
4) Receives results from collection servers
5) Merges and presents results to user
Step 2 is the Collection Selection (a/k/a Database Selection) Problem.
This is our focus.
TREC-Based Test Environment
• 6 Sources: AP, FR, PATN, SJM, WSJ, ZIFF
– Sources partitioned into a total of 236 collections (a/k/a sites, databases)
– Different decomposition (partitioning) methods
• 250 Queries– Often contain many [possibly repeating] query terms
• 1000s of Relevance Judgements– Humans listed documents relevant to each query– Complete only for queries 51-150
SYM Decomposition
• SYM = Source Year Month
• Example collections:– AP.88.02 - contains all Associated Press
documents from February 1988– ZIFF.90.12 - contains all ZIFF documents
from December 1990
• 236 collections total
SYM’s “Goto AP” Problem
• In SYM, larger collections (# documents) tend to contain more of the relevant documents
• Some collections (AP) so large w.r.t. others that any algorithm favoring large collections over small collections will perform well, independently of the query contents
UDC Decomposition
• UDC = Uniform Document Count
• Example collections:– AP.01 - contains the first 1/236 (approx) of the
complete set of documents; restricted to AP documents only [so no overlap w/ FR]
– ZIFF.45 - contains the last 1/236 (approx) of the complete set documents
• 236 collections total
Baselines
• RBR = Relevance Based Ranking– meritquery,coll = number of documents in collection
which were deemed relevant to query
• Others...
Estimates
• Estimate names/categories:– gGLOSS - ideal(k)– CORI - U.Mass, best performance– SMART - ntn_ntn, etc. (tweaked components)– CVV - Yuwono/Lee’s “traditional” version
– CVVp,q,r,s - hybrid, elements from CVV, SMART
– SBR, random, etc.
• meritquery,coll = estimated merit (“goodness”)
Ranks• For each query, collections are ranked
according to [estimated] merit (highest 1st)
• A ranking represents the order in which collections should be searched for docs
• Often want to consider only top-n collections– n is the “collection cut-off” or simply “cut-off”
Rank Comparisons
• An estimate’s ranks are compared against a baseline’s ranks (e.g., RBR vs CORI)
• Different comparison metrics:– Mean Squared Error (MSE), Spearman’s rho– P(n) = Precision at cut-off n
• prop. of estimated top-n collections with real merit > 0
– R(n) = Recall at cut-off n• prop. of top-n real merit in estimated top-n collections
– R^(n) = “R hat”(n) = H(n)• prop. of total real merit in estimated top-n collections
Overall Performance Metric
• In this study, we use R(avg) averaged over all queries
• This is a normalized “area under the curve”
What is CVV?
• CVV = Cue Validity Variance
• CVV algorithm– a/k/a “CVV ranking method”– an estimator, generates merits (which can be
ranked and compared against a baseline)– consists of CVV component, DF component
• CVV component– derived from DF and N
• What are DF and N?
Terminology, cont.
• C = set of collections in the system– |C| = number of collections in the system (236)
• Ncoll = number of documents in collection
• DFterm, coll = “Document Frequency”
– number of documents in collection in which term occurs at least once
– So, DFterm,coll <= Ncoll
Cue Validity (CV)
• Densityterm,coll = DFterm,coll / Ncoll
• Externalterm,coll = (DFterm,c) / (Nc)
c != coll c != coll
• CVterm,coll = Densityterm,coll
Densityterm,coll + Externalterm,coll
• CVterm = (CVterm,coll) / |C| coll in C
CVterm,coll values are always between 0 and 1
Cue Validity Variance (CVV)
• CVVterm = (CVterm,coll - CVterm)2 / |C| coll in C
• meritquery,coll = estimated “Goodness”
= (CVVterm * DFterm,coll) term in query
2 components
• The basic CVV algorithm ignores the number of times a term occurs in the query (Query Term Weight)
Basic CVV “Problem”
• What if query = “cat cat dog”?– Basic CVV ignores the number of times “cat”
appears in the query, so merits (and consequently ranks) will be the same as for query “cat dog”
– Unlike in CVV’s development environment, many of our test queries have multiply-occurring terms
– Performance of basic CVV tends to be noticeably poorer than that of algorithms which incorporate the query term frequency in their merit calculations
– We can modify CVV to incorporate query term frequency
Enchancing CVV with QTW
• QTWquery,term = query term weight (or freq) = # of times term occurs in query
• meritquery,coll = (CVVterm * DFterm,coll * QTWquery,term) term in query
3 components
Effects of QTW Enhancement• For R(avg) averaged over all queries…
– SYM performance increased from .8416 to .8486– UDC performance increased from .6735 to .6846– Meanwhile, CORI’s SYM = .8972, UDC = .7884– So, performance increased, but not dramatically
• For most individual queries, performance improved or stayed the same, but for 20-30% of queries, performance decreased– Could QTW be overcorrecting?
Further Questions...• What if we varied the degree to which QTW
influenced the merit calculations? We could:– Add an exponent to the QTW component:
meritterm,coll = (CVVterm * DFterm,coll * QTWterme)
– Evaluate using a range of exponents, searching for version that maximizes performance (e = 0, 0.5, 1, 2, …)
• What if we added exponents to the other two components as well: CVV and DF?
• What about adding an ICF component?
Inverse Collection Frequency (ICF)
• CFterm = collection frequency of term = # of collections in which term occurs
• ICFterm = log ((|C| + 1) / CFterm)
• ICF is used to decrease contribution of terms which appear in many collections and are therefore not good discriminators; terms which occur in few collections are best discriminators.
• ICF has shown useful in other algorithms.
Final Enhancement Equation
• meritquery,coll =
(CVVtermp
* DFterm,collq
* QTWquery,termr * ICFterm
s) term in query
4 components
• Notes:– DF is the only component dependent upon collection– CVV(1,1,0,0) = Basic CVV = CVV(0) or “cvv”– CVV(1,1,1,0) = QTW-Enhanced CVV = CVV(1)– CVV(0,1,1,2) = ntn_ntn– CVV(*,0,*,*) = alphabetical (same merits for all collections)
P E R F O R M A N C EEstimate SYM UDC Joint
CORI .8972 .7884 .8428
1.0 0.5 2.0 2.0 .8938 .7373 .8155
1.0 0.5 2.0 0.7 .8839 .7776 .8307
0.5 0.2 3.0 1.0 .8937 .7712 .8325
Basic CVV .8416 .6735 .7576
Ideal(0) .8570 .7146 .7858
ntn_ntn .8729 .7356 .8042
Note: ntn_ntn = 0.0 1.0 1.0 2.0
Essentiality• What’s the best performance you can get if you
hold a component’s exponent at 0?
• CVV component appears to be the least essential;DF appears to be the most essential
Omitted Best Performer and Performance Comp SYM UDC Joint . none 1.0 0.5 2.0 2.0 (.8938) 1.0 0.5 2.0 0.7 (.7776) 0.5 0.2 3.0 1.0 (.8325)
CVV 0.0 0.5 3.0 3.0 (.8932) 0.0 0.5 2.0 1.0 (.7737) 0.0 0.5 3.0 1.0 (.8298) DF tied (.6081) tied (.6017) tied (.6049) QTW 1.0 0.5 0.0 2.0 (.8851) 3.0 0.5 0.0 0.0 (.7525) 0.0 0.5 0.0 1.0 (.8053)
ICF 2.0 0.5 3.0 0.0 (.8755) 3.0 0.5 2.0 0.0 (.7647) 3.0 0.5 3.0 0.0 (.8188)
Other Reductions
Active Best Performer and Performance Comps SYM UDC Joint . all 1.0 0.5 2.0 2.0 (.8938) 1.0 0.5 2.0 0.7 (.7776) 0.5 0.2 3.0 1.0 (.8325)
DF,CVV 0.0 0.5 0.0 0.0 (.8533) 3.0 0.5 0.0 0.0 (.7525) 2.0 0.5 0.0 0.0 (.7945)
DF,QTW 0.0 0.5 3.0 0.0 (.8725) 0.0 0.5 3.0 0.0 (.7306) 0.0 0.5 3.0 0.0 (.8015)
DF,ICF 0.0 0.5 0.0 2.0 (.8831) 0.0 0.5 0.0 1.0 (.7372) 0.0 0.5 0.0 1.0 (.8053)
DF only 0.0 0.5 0.0 0.0 (.8533) 0.0 0.5 0.0 0.0 (.6889) 0.0 0.5 0.0 0.0 (.7711)
• Comments:– Fewer sample points, since we didn’t focus searches here
Open Questions• How much would other algorithms’
performances improve if we tweaked them?– Would CORI get so much better that a CVV
hybrid couldn’t even get close?– Or, is CORI already optimized?
• Is there value in automating these searches, using adaptive programming?
Basic CVV Example• Given:
– query1 = “cat dog”
– CVV“cat” = 0.8 (“cat” is unevenly distributed)
– CVV“dog” = 0.2 (“dog” is more evenly distributed)
– DF“cat”,collA = 1 (“cat” appears in only one of collA’s docs)
– DF“dog”,collA = 20
– DF“cat”,collB = 5
– DF“dog”,collB = 3
• meritquery1,collA = (0.8*1) + (0.2*20) = 0.8 + 4 = 4.8
• meritquery1,collB = (0.8*5) + (0.2*3) = 4 + 0.6 = 4.6
For query1, collA would be ranked higher (better) than collB