LIS618 lecture 4

Thomas Krichel

2004-02-28

Structure

• Document preprocessing

• Practice: Nexis

document preprocessing

• There are some operations that may be done to the documents before indexing– lexical analysis– stemming of words– elimination of stop words– selection of index terms– construction of term categorization structures

we will look at those in turn• in many cases, document preprocessing is not

well documented by the provider.• but searchers need to be aware of them…

lexical analysis

• divides a stream of characters into a stream of words

• seems easy enough but….– should we keep numbers?– hyphens. compare "state-of-the-art" with "b-52"– removal of punctuation, but “The battle of Cannae

took place in 217 B.C. It was a bad defeat for the Romans."

– casing. compare "bank" and "Bank"

stemming

• in general, users search for the occurrence of a term irrespective of grammar

• plural, gerund forms, past tense can be subject to stemming

• important algorithm by Porter, applicable to English only

• evidence about the effect of stemming on information retrieval is mixed

• stemming is relatively rare these days.

elimination of stop words

• some words carry no meaning and should be eliminated

• in fact any word that appears in 80% of all documents is pretty much useless, but

• consider a searcher for "to be or not to be".

• It is better to reduce the index weight of terms that appear very frequently

index term selection

• In printed indexes, we use nouns only• some nouns that appear heavily together can be

considered to be one index term, such as "computer science"

• Dialog deals with this through phrase indexing. • Nexis has the smart indexing feature that groups

terms into concepts• Most web engines, index all words, and all of the

individually

thesauri

• a list of words and for each word, a list of related words – synonyms– broader terms– narrower terms

• used– to provide a consistent vocabulary for indexing and

searching– to assist users with locating terms for query

formulation– allow users to broaden or narrow query

use of thesauri

• Thesauri are limited to experimental systems, or some high-quality systems, examples are

– http://www.sosig.ac.uk/roads/cgi-bin/thesaurus.pl

– Nexis

• It can be confusing to users.

• There is very little free thesaurus data available.

Back to Nexis: customization

• You can customize the search results on the top right corner of the screen.– You should set your default result list to the

expanded list, so you can see your keywords in context of the documents.

– You may wish to increase the size of the default page to 99.

• You can hide the subject directory.

test sources

• The ones that I have used are – News 60 days (the top source)– Two foreign sources

• Le Monde • Der Spiegel

dating

• Dates can be entered in any of the following forms– 07/24/2000 -- 7/24/00 – July 24, 2000 -- Jul 24, 2000 – July 2000

• You can set the dates in the menu, but they will not go beyond the stated frame of the source.

Document preprocessing in Nexis

• The following are always considered word limits– hyphens -– slashes / – parentheses ()– spaces

• examples– “state of the art” will also find “state-of-the-art”– “co-operative” will not find “cooperative”.

• The documentation says you have to leave them out of the query. I think this is wrong, you can use slash and hyphen.

query preprocessing in Nexis

• apostrophe: – if followed by "s", it is a possessive. Singular and

plural and plural possessive are also found. Thus “company’s” also finds

• company• companies• companies’

– if not, it counts like a character in a word

• at-sign: contrary to documentation, does not appear to be preprocessed. But you can leave it out when you search for an email addresses. “president@whitehouse.gov” gives the same results as “president whitehouse.gov”

query preprocessing in Nexis

• ampersand: if it is surrounded by blanks, most of the time, Nexis treats it as "and". If it is not, it treats it as a normal character

• But the & is not always equivalent to “and”.

• Example: search in Le Monde for – “cable & wireless” same as “cable wireless”– “cable and wireless” gives a different result

query preprocessing in Nexis

• colon and comma are read as a space unless adjacent characters are numbers.

• percent and pound sign mean themselves and are not equivalent to anything.

• ? $ ; in the query are all ignored, it is said in the documentation, but that is not true– “$ 4711” in Der Spiegel versus “4711” in the same

source

• ® is replaced by the word "R", ™ is replaced by the word "TM“, according to documentation.

query preprocessing in Nexis

• if you enter a dot, it is interpreted as a decimal point if surrounded by numbers

• if the dot is followed by just one letter, you have to keep it – “10:14 a.m.” will get some results– “10:14 am” will get different results

• but if there are several letters, you can leave it out– “ebay com” gives same results as “ebay.com”

query preprocessing in Nexis

• The double quote is ignored in a query. • It can be used to make the word not, a

reserved word, searchable, according to the documentation.

• Example– “to be or not to be”, Der Spiegel, all dates

• This method can not be applied to other noise words. It is not even possible to get lists of noise words.

effort to be easy

• Overall, Nexis makes an effort to be easy at the expense of precise query semantics.

• One positive example are long quotes– who said: “For the past three years there has been no

growth. Sooner or later they have to figure out that the melody has changed.”

– Such long quotes work most of the time, they can be entered without surrounding quotation marks.

noise and reserved words

• Noise words are common words– in power search, noise words are ignored, replace by

space– in quick search, you can use phrases– no list of noise words

• Reserved words are – and– or– not

used in Boolean expressions. They are not indexed.

plurals

• Nexis indexes plural and possessive as the singular.

• But in power search, you can use the following– PLURAL (term) only the plural of term– SINGULAR (term) only the singular of term– ALLCAPS (term) only capitals of term– NOCAPS (term) no capitals of term– CAPS (term) capitalized term only

• Note that term can be a sequence of words.• This feature does not work properly on certain

databases. Example allcaps(RFA) in Le Monde.

plural, singular etc functions

• Such functions are not reliable. Example, search in the Der Spiegel– national plural(archives) no results– national archives– singular(haus) same results as “haus”– plural(haus) no results– upper(USA) no results

entering phrases in power search

• If you just put one after the other, it looks for the one after the other subject to– latin chars only

• Путин does not work– no hits (opera)– erroneous hits (explorer)

– removal of noise words• “Thomas Bishoff” finds Thomas of Bishoff

– plural ORing• “José Carreras” will also find José Carrera

searching for phrases

• If your phrase contains a word that is a reserved word you are best off using connectors.

• Example, current news, today only– “black white” 221 hits

• “black and white”• “black or white”• “black/white”

– “black and white” 444 hits• interpreted “and” as a reserved word.

searching for phrases

• If your phrase contains a word that likely to be a word you are best off using connectors.

• Otherwise, you search seems to be translated with the noise word removed and then replaced by a connector.

• Example, current news, 60 days– “cream of white” you get

• “cream or white”• “cream/white”• “cream into the white”

treatment of accents

• I have done a lot of research, but found no systematic treatment of accents. It seems source dependent.

• Therefore search several variations, e.g. – Müntefering – Muentefering – Muntefering

• Same thing for plurals. Some foreign language sources are not searched for plurals. Example, search “archive” in Der Spiegel.

order of connectors• The order is

– OR – W/n, PRE/n – W/S (can not be combined with w/n or pre/n)– W/P (can not be combined with w/n or pre/n)– AND – AND NOT

• When you use two or more of the same connectors in a search, they normally operate from left to right.

• When a search contains multiple W/n or PRE/n connectors, the connectors operate in numerical order with the smallest number first.

examples

• use news most current 60 days– “Harry pre/1 Miller” 18 hits

• 5th has the sentence “HAWTHORN has discovered another Harry Miller”.

– “Harry pre/2 Miller” 93 hits

• “another pre/2 Miller w/3 Hawthorn” matches “HAWTHORN has discovered another Harry Miller”, because the pre/2 is evaluated first.

problem formats

• Sometimes Nexis gets confused!• Problem Format:          Corrected Format:

– A w/n (B and C) A w/n B and A w/n C– A w/s (B and C) A w/s B and A w/s C– A w/p (B and C) A w/p B and A w/p C– A w/s (B and C) A w/s B and A w/s C

• but of course you can do something like “another pre/2 Miller and Miller w/s Hawthorn”

Nexis segments

• Nexis does some document preprocessing for characters, discussed in a later slide.

• The processed document has a number field/value pairs that are called segments

• Not every source has every segment.

• Searches using segments will return no hits for sources that don’t have the segment.

segment types

• I make a distinction between – native – smart-indexed

segments.

• Smart-indexed segments contain the result of smart indexing. Thus, they are not native to the source.

segment types

• Some segment can be sorted. There are the ones that have numbers or dates as values.

• These segments use the following arithmetic operators:– = is equal to or is– > aft greater than or after– < bef less than or before

• Example for Der Spiegel• atleast3(foster) and date(<2000) and

length(>1000)

how to know about segment

• This is virtually impossible, because there is no comprehensive documentation.

• Look at the example of Le Monde, where the list of segments seems incomplete.

• For Der Spiegel “rubrik(hausmitteilung) and text(foster)” does find the example given. “section(hausmitteilung) and body(foster)”

typical segments in news

• BYLINE • CORRRECTION • CORRECTION-DATE (date)• DATE (date)• DATELINE (not a date) • GRAPHIC• HEADLINE • HIGHLIGHT • LEAD• HLEAD is HEADLINE, HIGHLIGHT, & LEAD

typical segments in news

• PUBLICATION name and copyright

• SECTION

• SERIES

• SOURCE

• TICKER  

• TYPE

typical smart-indexed segments

• CITY• COMPANY • COUNTRY • GEOGRAPHIC • INDUSTRY• KEYWORD • ORGANIZATION• PERSON • PRODUCT• SUBJECT • TICKER• TYPE• TERMS includes all these

terms example

• compare, in recent news, person(osama bin laden) with “osama bin laden”

• index terms can be useful to hone in on complex topics that can have many names. Example– start with “sex” as an index term– collect terms related to gay marriage and civil

unions

segment search

• You can place query terms and connectors in a segment and then search for it. Example:

hlead((drug or substance) w/10 abuse)

using segments for news

• uses power search expressions, plus• hlead (expression) ?• headline (expression)• company (expression) for a company• byline (expression) for the author• show (expression) for a television show

transcript

expression is a simple keyword or expression using several words, possible combined with connectors.

segments for legal data

• name (expression) for the name of a party

• cite (expression) for a citation expression for case law

• title (expression) for the title of a law article

expression is a simple keyword or expression using several words, possible combined with connectors.

Search forms

• There are special forms for – News– Company reports– Market indicators– Portfolio– News and quotes about companies

Personal news alert

• do a search

• then click on “track in personal news” to get to a screen where you can enter – periodicity – what documents to be sent– subject

• This works for real estate for me.

Real time news

• This uses a different query language– terms are implicitly ANDed– explicit AND and OR allowed– phrases have to be put in quotes– * starts for any number of characters, not just

one as in power search– parenthesis can be used

• I have poor experience with this.

Summary on Nexis

• Nexis has a rich set of resources.

• It can be searched by inexperienced, but likely to get poor result.

• Clever learning about its features can get you quite far, however, the features are not well documented online.

• Nexis seems frequently to violate its stated rules.

http://openlib.org/home/krichel

Thank you for your attention!