Upload
rosa-beverley-robinson
View
233
Download
0
Embed Size (px)
Citation preview
Lucene
Open Source Search Engine
Lucene - Overview
• Complete search engine in a Java library• Stand-alone only, no server– But can use SOLR
• Handles indexing and query• Fully featured – but not 100% complete• Customizable – to an extent• Fully open source• Current version: 3.6.1
Lucene Implementations• LinkedIn
– OS software on integer list compression• Eclipse IDE
– For searching documentation• Jira• Twitter• Comcast
– XfinityTV.com, some set top boxes• Care.com• MusicBrainz• Apple, Disney• BobDylan.com
Indexing
Lucene
Lucene - Indexing• Directory = A reference to an Index– RAMDirectory, SimpleFSDirectory
• IndexWriter = Writes to the index, options:– Limited or unlimited field lengths– Auto commit– Analyzer (how to do text processing, more on this later)– Deletion Policy (only for deleting old temporary data)
• Document – Holds fields to index• Field – A name/value pair + index/store flags
Lucene – Indexer OutlineSimpleFSDirectory fsDir = new SimpleFSDirectory(File)IndexWriter iWriter = new IndexWriter(fsDir,…)Loop: fetch text for each document {
Document doc = new Document();doc.add(new Field(…)); // for each fieldiWriter.addDocument(doc);
}iWriter.commit();iWriter.close();fsDir.close();
Class Materials
• SharePoint link– use “search\[flast]” username– sharepoint.searchtechnologies.com– Annual Kickoff– Shared Documents– FY2013 Presentations– Introduction to Lucene
• lucene-training-src-FY2013.zip
Lucene – Index – Exercise 1• Create A new Maven Project
– mvn archetype:generate -DgroupId=com.searchtechnologies -DartifactId=lucene-training -DarchetypeArtifactId=maven-archetype-quickstart -DinteractiveMode=false
– Right click pom.xml, Maven.. Add Dependency• lucene-core in search box• Choose 3.6.1• Expand Maven Dependencies.. Right click lucene-core.. Maven download
sources
– Source code level = 1.6• Copy Source File: LuceneIndexExercise.java
– Into com.searchtechnologies package• Copy data directory to your project• Follow instructions in the file
Query
Lucene
Lucene - Query• Directory = An index reference• IndexReader = Reads the index, typically
associated with reading document fields– readOnly
• IndexSearcher = Searches the Index• QueryParser – Parses a string to a Query– QueryParser = Standard Lucene Parser– Constructor: Version, default field, analyzer
• Query – Query expression to execute– Returned by qParser.parse(String)– Search Tech’s QPL can generate Query objects
Lucene – Query part 2• Executing a Search– TopDocs td =
iSearcher.search(<query-object>, <num-docs>)• TopDocs – Holds statistics on the search plus the
top N documents– totalHits, scoreDocs[], maxScore
• ScoreDoc –Information on a single document– Doc ID and score
• Use IndexReader to fetch any Document from a Doc ID– (includes all fields for the document)
Lucene – Search OutlineSimpleFSDirectory fsDir = new SimpleFSDirectory(File f)IndexReader iReader = new IndexReader(fsDir,…)IndexSearcher iSearcher = new IndexSearcher(iReader)StandardAnalyzer sa = new StandardAnalyzer(…)QueryParser qParser = new QueryParser(…)Loop: fetch a query from the user {
Query q = qParser.parse( <query string> )TopDocs tds = iSearcher.search(q, 10);
Loop: For every document in tds.scoreDocs { Document doc = iReader.document(tds.scoreDocs[i].doc);
Print: tds.scoreDocs[i].score, doc.get(“field”)}
}// Close the StandardAnalyzer, iSearcher, and iReader
Lucene – Query – Exercise 2
• Open Source File: LuceneQueryExercise.java• Follow instructions in the file
Relevancy Tuning
Lucene
Lucene Extras – Fun Things You Can Do
• iWriter.updateDocument(Term, Document)– Updates a document which contains the “Term”– “Term” in this case is a field/value pair• Such as “id” = “3768169”
• doc.boost( <float boost value> )– Multiplies term weights in the doc by boost value– Part of “fieldNorm” when you do an “explain”
• field.boost( <float boost value> )– Multiplies term weights in field by boost value
Explain - Example
iSearcher.explain(query, doc-number)
Query: star OR catch^0.6 for document 903
1.2778056 = (MATCH) product of: 2.5556111 = (MATCH) sum of: 2.5556111 = (MATCH) weight(title:catch^0.6 in 903), product of: 0.56637216 = queryWeight(title:catch^0.6), product of: 0.6 = boost 7.2195954 = idf(docFreq=1, maxDocs=1005) 0.13074881 = queryNorm 4.512247 = (MATCH) fieldWeight(title:catch in 903), product of: 1.0 = tf(termFreq(title:catch)=1) 7.2195954 = idf(docFreq=1, maxDocs=1005) 0.625 = fieldNorm(field=title, doc=903) 0.5 = coord(1/2)
Lucene – Query– Exercise 3
• Add explain to your query programExplanation exp = iSearcher.explain( . . . )
• Call it for all documents produced by your search
• Simply use toString() on the result of explain() to display the results
Boosting – Other Issues
• Similarity Class Javadoc Documentation– Very useful discussion of boosting formulas
• Similarity.encodeNormValue() – 8-bit floating point!
0.00 => 00.10 => 6E0.20 => 720.30 => 740.40 => 760.50 => 780.60 => 780.70 => 790.80 => 7A0.90 => 7B1.00 => 7C1.10 => 7C1.20 => 7C
1.30 => 7D1.40 => 7D1.50 => 7E1.60 => 7E1.70 => 7E1.80 => 7F1.90 => 7F2.00 => 802.10 => 802.20 => 802.30 => 802.40 => 802.50 => 80
2.60 => 812.70 => 812.80 => 812.90 => 813.00 => 813.10 => 823.20 => 823.30 => 823.40 => 823.50 => 823.60 => 833.70 => 833.80 => 83
3.90 => 834.00 => 834.10 => 844.20 => 844.30 => 844.40 => 844.50 => 844.60 => 844.70 => 844.80 => 844.90 => 845.00 => 84
Lucene Query Objects
• Query objects are used to execute the search
QueryParser
QueryString
iSearcher.search()
TopDocs
All Derived from the Lucene Query class
Lucene Query Objects - Example
(george AND washington) OR (thomas AND jefferson)
BooleanQuery(clauses = SHOULD)
BooleanQuery(clauses = MUST)
TermQuerygeorge
TermQuerywashington
TermQuerythomas
TermQueryjefferson
BooleanQuery (clauses = MUST)
Lucene BooleanQuerygeorge +washington -martha jefferson -thomas +sally
WORKS LIKE AND:
BooleanQuery bq = new BooleanQuery();bq.add( X , Occur.MUST);bq.add( Y , Occur.MUST);
WORKS LIKE OR:
BooleanQuery bq = new BooleanQuery();bq.add( X , Occur.SHOULD);bq.add( Y , Occur.SHOULD);
WORKS LIKE: X AND (X OR Y)
BooleanQuery bq = new BooleanQuery();bq.add( X , Occur.MUST);bq.add( Y , Occur.SHOULD);
Lucene – Query– Exercise 4• Create BooleanQuery and TermQuery objects as
necessary to create a query without the query parser• Goal: (star AND wars) OR (blazing AND saddles)• TermQuery:
tq = new TermQuery(new Term("field","token"))
• BooleanQuery:BooleanQuery bq = new BooleanQuery();bq.add( <nested query object> , Occur.MUST);bq.add(<nested query object> , Occur.MUST);
• Occur– Occur.MUST, Occur.SHOULD, Occur.MUST_NOT
• TermQuery and BooleanQuery derive from Query– Any “Query” objects can be passed to iSearcher.search()
Lucene Proximity Queries• “Spanning” Queries Return matching “spans”
DOCUMENT: Four score and seven years ago, our forefathers brought forth…
Query: Returns:
four before/5 seven 0:4
(four before/5 seven) before forefathers 0:8
brought near/3 ago 5:9
(four adj score) or (brought adj forth) 0:2, 8:10
0 1 2 3 4 5 6 7 8 9 10
Word positions mark word boundaries
Proximity Queries : Available Operators
• (standard) SpanTermQuery– For terms inside spanning queries
• (standard) SpanNearQuery– Inorder flag handles both near and before
• (standard) SpanOrQuery• (standard) SpanMultiTermQueryWrapper– fka SpanRegexQuery
• (Search Tech) SpanAndQuery• (SearchTech) SpanBetweenQuery– between(start,end,positive-content,not-content)
Span Queries
• demo of LuceneSpanDemo.java
Analysis
Lucene
Analyzers• “Analysis” = “Text Processing” in Lucene• Includes:– Tokenization
• Since 1955, the B-52… Since, 1955, the, B, 52
– Token filtering• Splitting, joining, replacing, filtering, etc.• Since, 1955, the, B, 52 1955, B, 52• George, Lincoln george, lincoln• MasteringBiology Mastering, Biology• B-52 B52, B-52, B, 52
– Stemming• tables table• carried carry
Analyzer
Analyzer, Tokenizer, TokenFilter
• Tokenizer: Text TokenStream• TokenFilter: TokenStream TokenStream• Analyzer: A complete text processing function
(one tokenizer + multiple token filters)– Manufactures TokenStreams
Tokenizer TokenFilter TokenFilter . . .string
Existing Analyzers, Tokenizers, Filters
• Tokenizer– (Standard) CharTokenizer, WhitespaceTokenizer,
KeywordTokenizer, ChineseTokenizer, CJKTokenizer, StandardTokenizer, WikipediaTokenizer (more)
– (Search Tech) UscodeTokenizer (produces each HTML <tag> as a separate token)
• TokenFilter– Stemmers: (Standard) many language-specific
stemmers, PorterStemFilter, SnoballFilter– Stemmers: (Search Tech) Lemmatizer
Existing Analyzers, Tokenizers, Filters
• TokenFilters (continued)– LengthFilter, LowerCaseFilter, StopFilter,
SynonymTokenFilter (don’t use), WordDelimiterFilter (SOLR only)
• Analyzers– WhitespaceAnalyzer, StandardAnalyzer, various
language analyzers, PatternAnalyzer
Analyzers almost always need to be customized.
Creating and Using TokenStreamTokenStream tokenStream = new SomeTokenizer(…);tokenStream = new SomeTokenFilter1(tokenStream);tokenStream = new SomeTokenFilter2(tokenStream);
CharTermAttribute charTermAtt =tokenStream.getAttribute(CharTermAttribute.class);
OffsetAttribute offsetAtt =tokenStream.getAttribute(OffsetAttribute.class);
while (tokenStream.incrementToken()) {charTermAtt Now contains info on the token’s termoffsetAtt.startOffset() Now contains the token’s start offset
}
Token Streams - How They Work
TokenFilterCall incrementToken()
TokenFilter
Call incrementToken()
Tokenizer
Call incrementToken()
Get next tokenfrom Reader() store in
Attribute objects
Return
Modify attribute objectsand return
Modify attribute objectsand return
Use Attribute Objects
Creating and Using TokenStream
DEMO
Replacement Pattern
TokenFilter
incrementToken()
Call incrementToken()
Modify attribute objectsand return
Token Filters Simply Modify Attributes that Pass Through
Token Filter – Replacement Patternpublic final class LowerCaseFilter extends TokenFilter { public LowerCaseFilter(TokenStream input) { super(input); termAtt = (CharTermAttribute)
addAttribute(CharTermAttribute.class); }
private CharTermAttribute termAtt;
public final boolean incrementToken() throws IOException { if (input.incrementToken()) { final char[] buffer = termAtt.buffer(); final int length = termAtt.length();
for(int i=0;i<length;i++) buffer[i] = Character.toLowerCase(buffer[i]);
return true; } else return false; }}
Deletion Pattern
TokenFilter
incrementToken()
Call incrementToken()
Token Filters Check Token Attributes and May Call incrementToken() Multiple Times
Keep Looping UntilA Good Token is Found
Then Return It
Token Filter – Deletion Patternpublic final class TokenLengthLessThan50CharsFilter extends TokenFilter { public TokenLengthLessThan50CharsFilter(TokenStream in) { super(in); termAtt = (CharTermAttribute) addAttribute(CharTermAttribute.class); posIncrAtt = (PositionIncrementAttribute) addAttribute(PositionIncrementAttribute.class); }
private CharTermAttribute termAtt; private PositionIncrementAttribute posIncrAtt;
public final boolean incrementToken() throws IOException { int skippedPositions = 0; while(input.incrementToken()) { final int length = termAtt.length();
if(length > 50) { skippedPositions += posIncrAtt.getPositionIncrement(); continue; }
posIncrAtt.setPositionIncrement(posIncrAtt.getPositionIncrement() + skippedPositions);
return true; } return false; }}
Splitting Tokens Pattern – First Call
TokenFilter
incrementToken()
Call incrementToken()
When Splitting a Token, Save the Splits Aside For Later
Saved Token
Split Token
ReturnFirstHalf
SaveSecond
Half
Splitting Tokens Pattern – Second Call
TokenFilter
incrementToken()
When Called the Second Time, Just Return Saved Token
Saved Token
Return Saved Token
Token Filter – Splitting Patternpublic final class SplitDashFilter extends TokenFilter { public SplitDashFilter(TokenStream in) { super(in); termAtt = (CharTermAttribute) addAttribute(CharTermAttribute.class); } private CharTermAttribute termAtt; char[] saveToken = new char[100]; // Buffer to hold tokens from previous incrementToken() call int saveLen = 0;
public final boolean incrementToken() throws IOException { if(saveLen > 0) { // Output previously saved token termAtt.setEmpty(); termAtt.append(new String(saveToken, 0, saveLen)); saveLen = 0; return true; } if (input.incrementToken()) { // Get a new token to split final char[] buffer = termAtt.buffer(); final int length = termAtt.length(); boolean foundDash = false; for(int i=0;i<length;i++) { // Scan token looking for ‘–’ to split it if(buffer[i] == ‘-’) { foundDash = true; termAtt.setLength(i); // Set length so termAtt = first half now } else if(foundDash) saveToken[saveLen++] = buffer[i]; // Save second half for later } return true; // Output first half right away } else return false; }}
Token Splitting
DEMO
Stemmers and Lemmatizers• Stemmers available in Lucene
– Snoball, Porter– They are both terrible [much too aggressive]– For example: mining min
• Kstem– Publicly available stemmer with Lucene TokenFilter
Implementation– Better, but still too aggressive:
• searchmining searchmine
• Search Technologies Lemmatizer– Based on GCIDE Dictionary– Extremely accurate, only reduces words to dictionary entries– Also does irregular spelling reduction: mice mouse– STILL A WORK IN PROGRESS: Needs one more refactor
ST Query Processing
Lucene
Search Technologies Query Parser
• Originally written for GPO– Query FAST FQL
• Converted to .Net for CPA• Refactored for Lucene for Aspermont• Refactored to be more componentized and
pipeline-oriented for OLRC
• Still a work in progress– Lacks documentation, wiki, etc.
Search Technologies Query Processing
• Query Parser– Parses the user’s entered query
• Query Processing Pipeline– A sequence of query processing components which can
be mixed and matched• Lucene Query Builder• Other Query Builders Possible– FAST, Google, etc.– No others implemented yet
• Query Configuration File– Holds query parsing and processing parameters
Our Own Query Processing: Why?
• Gives us more control– Can exactly meet user’s query syntax
• Exposes operators not available through Lucene Syntax– Example: before proximity operator
• “behind the scenes” query tweaking– Field weighting– Token merging: rio tinto url:riotinto– Exact case and exact suffix matching– True lemmatization (not just stemming)
ST Query Parser – Overall Structure
Parser
QueryString
ProcessorTopDocs
Generic “AQNode”Structures
Processor LuceneBuilder
. . .
Lucene QueryStructures
The Search Technologies Query Structure
userQuery
nodeQuery
finalQuery
QueryString
Lucene QueryStructures
Generic AQNodeStructures
• Holds references to all query representations• Therefore, query processors can process any
query representation• Everything is a QueryProcessor– Parsing, processing, and query building
Query Parser: Features
• AND, OR, NOT, parenthesis– ((star and wars) or (star and trek))– star and not born {broken}
• +, -+ = query boost- = not {broken}
• Proximity operators– within/3, near/3, adj
• Phrases• field: searches• title:(star and wars) and description:(the original)
Using Query Processors
• Load Query ConfigurationQueryConfig qConfig = new QueryConfig("data/query-config.xml");
• Create Query ProcessorIQueryProcessor iqp2 = new TokenizationQueryProcessor();
• Initialize Query Processoriqp2.initialize(qConfig);
• Use Query Processors (simply call in sequence)iqp1.process(query);iqp2.process(query);iqp3.process(query);
Query Processors: Other Notes
• Types of processors: (off the shelf)– Lemmatizer, tokenization, lower case
• QueryParser and Query classes may need to be fully qualifiedcom.searchtechnologies.queryprocessor.Query query = new com.searchtechnologies.queryprocessor.Query(queryString);
• Query Parser Only Splits on Whitespacestar-trek or star-wars or(star-trek,star-wars)
• Use TokenizationQueryProcessor to split fully or(phrase(star,trek),phrase(star,wars))
ST Query Processor – Exercise 5
• Add ST QueryProcessor to your Lucene Query• Add Dependency to your pom.xml:– com.searchtechnologies: st-queryparser: 0.3
• Add Processors– com.searchtechnologies.queryprocessor.QueryParser– TokenizationQueryProcessor()– LowercaseQueryProcessor()– LuceneQueryBuilder()
• Initialize Config, Construct Processors, Initialize Processors, Execute Processors
Creating Your Own Query Processor
• AQNode – “Aspire Query Node”– Operands – list of operands (references to other
AQNodes)– Operator – Enumerated list (AND, OR, NEAR…)– Proximity window (int)– From value, to value (objects)
• Use from value for token strings• Use from + to value for date ranges, int ranges, etc.
– startChar, endChar (in original user’s query string)– Enclosing field name– Other stuff for future expansion
• Attached data objects• Custom Query Builder
Query Processor Outline
public class MyQueryProcessor implements IQueryProcessor {
@Override public void initialize(QueryConfig config) throws QueryProcessorException { // Read any parameters you need from the config // config is an AXML (a wrapper around a W3C DOM object) }
@Override public void process(Query query) throws QueryProcessorException { // Process the query // query.getNodeQuery() The AQNode version of the query // query.getUserQuery() The original query string // query.getFinalQuery() The final (typically Lucene) query structure }}
Query Processor Examplepublic class LowercaseQueryProcessor implements IQueryProcessor {
@Override public void initialize(QueryConfig config) throws QueryProcessorException { }
@Override public void process(Query query) throws QueryProcessorException { convertToLowerCaseAQNodes(query.getNodeQuery()); } void convertToLowerCaseAQNodes(AQNode aqn) { if(aqn.getOperator() == AQNode.OperatorEnum.TERM) { String termText = (String)aqn.getFromValue(); aqn.setFromValue(termText.toLowerCase()); return; }
if(aqn.getOperands() == null) return;
for(AQNode childAqn : aqn.getOperands()) { convertToLowerCaseAQNodes(childAqn); } }}
ST Query Processor – Exercise 6
• Copy the FixStarQueryProcessor– Looks for “sta” and changes them to “star”
• Fill out the contents of the QueryProcessor• Add the QueryProcessor to your query
program• Run the program and query on “sta”– Add to STQueryProcessorExcercise5.java
Query Processing New Features
• Template Substitution (OLRC)– field:() searches are substituted for arbitrary query
expressions• Lemmatization (OLRC, BNA)• Wildcard Handling (OLRC)• Refactor Aspermont Query Processors– Semantic Network Expansion (ontology)– Add boost/reduce tokens (field:HI, field:LO)– Proximity boost– Composite fields and query field boost
Custom Hit Collector
• Collect() method called for each matching doc– Should be fast– Throw exception to break out of loop– Relation to Scorer
• DecadesCollector– Custom collector to take the top scoring document
from each decade• One main collector that wraps one
TopDocsScoreCollector per decade• See Source DecadesCollector.java
Complete
Open Source Search Engine