View
225
Download
0
Category
Preview:
Citation preview
1
Conventional Text-Retrieval Systems
Automatic Text Processing
by G. Salton, Addison-Wesley, 1989.
(Chapter 9)
2
Database Management
A specified set of attributes is used to characterize each record.EMPLOYEE(NAME, SSN, BDATE, ADDR, SEX, SALARY, DNO)
Exact match between the attributes used inquery formulations and those attached to the document.
SELECT BDATE, ADDRFROM EMPLOYEEWHERE NAME = ‘John Smith’
3
Text-Retrieval Systems
Content identifiers (keywords, index terms, descriptors) characterize the stored texts.
Degrees of coincidence between the sets of identifiers attached to queries and documents
content analysisquery formulation
4
Possible Representation
Document representation» unweighted index terms (term vectors)» weighted index terms» …
Query» unweighted or weighted index terms» Boolean combinations (or, and, not)» …
Search operation must be effective
5
File Structures
Main requirements» fast-access for various kinds of searches» large number of indices
Alternatives» Inverted Files» Signature Files» PAT trees
6
Inverted Files
File is represented as an array of indexed documents.
Term 1 Term 2 Term 3 Term 4
Doc 1 1 1 0 1
Doc 2 0 1 1 1
Doc 3 1 0 1 1
Doc 4 0 0 1 1
7
Inverted-file process
The document-term array is inverted (transposed).
Doc 1 Doc 2 Doc 3 Doc 4
Term 1 1 0 1 0
Term 2 1 1 0 0
Term 3 0 1 1 1
Term 4 1 1 1 1
8
Inverted-file process (Continued)
Take two or more rows of an inverted term-document array, and produce a single combined list of document identifiers.
Ex: Query= (term2 and term3)
term2 1 1 0 0term3 0 1 1 1------------------------------------------------------
1 <-- D2
9
List-merging for two ordered lists
The inverted-index operations to obtain answers are based on list-merging process.
ExampleT1: {D1, D3}T2: {D1, D2}Merged(T1, T2): {D1, D1, D2, D3}
10
Extensions of Inverted Index Operations(Distance Constraints)
Distance Constraints» (A within sentence B)
terms A and B must co-occur in a common sentence
» (A adjacent B)terms A and B must occur adjacently in the text
11
Extensions of Inverted Index Operations(Distance Constraints)
Implementation» include term-location in the inverted indexes
information: {P345, P348, P350, …}retrieval: {P123, P128, P345, …}
» include sentence-location in the indexes
information:{P345, 25; P345, 37; P348, 10; P350, 8; …}
retrieval:{P123, 5; P128, 25; P345, 37; P345, 40; …}
12
Extensions of Inverted Index Operations(Distance Constraints)
» Include paragraph numbers in the indexessentence numbers within paragraphsword numbers within sentencesinformation: {P345, 2, 3, 5; …}retrieval: {P345, 2, 3, 6; …}
» Query examples(information adjacent retrieval)(information within five words retrieval)
» Cost: the size of indexes
13
Term Weights
Term WeightsDi={Ti1, 0.2; Ti2, 0.5; Ti3, 0.6}
Issues» how to generate the term weights» how to apply the term weights
– Sum the weights of all document terms that match the given query.
– Rank the output documents in the descending order of term weight.
14
Boolean Query with Term Weights
Transform a Boolean expression into disjunctive normal form.
T1 and (T2 or T3)= (T1 and T2) or (T1 and T3)
For each conjunct, compute the minimum term weight of any document term in that conjunct.
The document weight is the maximum of all the conjunct weights.
15
Boolean Query with Term Weights
Example: Q=(T1 and T2) or T3Document Conjunct QueryVectors Weights Weight
(T1 and T2) (T3) (T1 and T2) or T3D1=(T1,0.2;T2,0.5;T3,0.6)
0.2 0.6 0.6D2=(T1,0.7;T2,0.2;T3,0.1)
0.2 0.1 0.2D1 is preferred.
16
Synonym Specification
Original Query(T1 and T2) or T3
Assume S1 is a synonym of T1.Assume S3 is a synonym of T3.
Broader Query((T1 or S1) and T2) or (T3 or S3)
The number of relevant items retrieved may be larger.
17
Stemming
Term Truncation» Remove suffixes and/or prefixes from context
terms.» Example
PSYCH*: psychiatrist, psychiatry, psychiatric,psychology, psychological, …
18
Term Truncation
Implementation» Only suffix truncation
Conventional inverted-index methodology can be maintained unchanged.
» Only prefix truncationThe term entries in inverted index are inversely alphabetized.antisymmetry --> yrtemmysitna
19
Term Truncation
» Both prefix and suffix truncation*SYMM*: antisymmetric, asymmetry inverted-index entries that are alphabetized both forward and backward
» infix truncationwom*n woman womeninverted index with entries for all possible “rotated” word forms
20
Term Truncation
Each term entry X=x1, x2, …, xn with individual characters xi is augmented by adding a special terminal character /.
ABC ABC/BABC BABC/BCAB BCAB/
Each augmented term x1, x2, …, xn/ is rotated cyclically by wrapping the term around itself n+1 times.
ABC / / ABC , C/ AB, BC/ A, ABC/
21
Term Truncation
Each resulting word form is then augmented by appending a blank character ^.
The resulting file of word forms is sorted alphabetically.^, /, a, b, c, …, Z
low high
ABC ABC/ /ABC^ /ABC^C/AB^ /BABC^BC/A^ /BCAB^ABC/^ AB/BC^
BABC BABC/ /BABC^ ABC/^C/BAB^ ABC/B^BC/BA^ B/BCA^ABC/B^ BABC/^BABC/^ BC/A^
BCAB BCAB/ /BCAB^ BC/BA^B/BCA^ BCAB/^AB/BC^ C/AB^CAB/B^ C/BAB^BCAB/^ CAB/B^
23
Retrieval Strategies
Query term XLook for index entries /X^ or X/^.
Query term X*Look for /X*.
Query term *XLook for X/^ => X/Y1, …, X/Yn.original patterns: X, Y1X, …, YnX
Query term *X*Look for XY1/Z1, …, XYn/Zn.original patterns: Z1XY1, …, ZnXYn
ABC ABC/ /ABC^ /ABC^ *B*C/AB^ /BABC^BC/A^ /BCAB^ABC/^ AB/BC^
BABC BABC/ /BABC^ ABC/^C/BAB^ ABC/B^BC/BA^ B/BCA^ BCABABC/B^ BABC/^ BABCBABC/^ BC/A^ ABC
BCAB BCAB/ /BCAB^ BC/BA^ BABCB/BCA^ BCAB/^ BCABAB/BC^ C/AB^CAB/B^ C/BAB^BCAB/^ CAB/B^
25
Retrieval Strategies
Query term X*YLook for Y/XZ1, …, Y/XZm.Original patterns: XZ1Y, …, XZmY
CostIncrease index entries.
Recommended