Numeric Range Queries in Lucene and Solr

Numeric Range Queriesin Lucene and Solr

kirilchukvadim@gmail.com

Agenda:

● What is RangeQuery

● Which field type to use for Numerics

● Range stuff under the hood (run!)

● NumericRangeQuery

● Useful links

Agenda:

● What is RangeQuery

● Which field type to use for Numerics

● Range stuff under the hood (run!)

● NumericRangeQuery

● Useful links

Range Queries:

A range query is a type of query that matches all documents where some value is between an

upper and lower boundary:

Give me:● Jeans with price from 200 to 300$● Car with length from 5 to 10m● ...

Range Queries:

In solr range query is as simple as:

q = field:[100 TO 200]

We will talk about Numeric Range Queriesbut you can use range queries for text too:q = field:[A TO Z]

Agenda:

● What is RangeQuery

● Which field type to use for Numerics

● Range stuff under the hood (run!)

● NumericRangeQuery

● Useful links (relax)

Which field type?

Which field type to use for “range” fields (let’s stick with int) in schema?

● solr.IntField● or maybe solr.SortableIntField● or maybe solr.TrieIntField

Which field type?

Let’s assume we have:● 11 documents, id: 1,2,3,..11● each doc has single value “int” price field ● document id is the same as it’s price● q = *:*

"numFound": 11, "docs": [ { "id": 1, “price_field": 1 },

{ "id": 2, “price_field": 2 }, ... { "id": 11, “price_field": 11 }]

Which field type - solr.IntField

q = price_field:[1 TO 10]

Which field type - solr.IntField

q = price_field:[1 TO 10]"numFound": 2, "start": 0, "docs": [ { "price_field": 1 }, { "price_field": 10 } ] }

Which field type - solr.IntField

Store and index the text value verbatim and hence don't correctly support range queries, since the lexicographic ordering isn't equal to

the numeric ordering

[1,10],11,2,3,4,5,6,7,8,9

Interesting, but “sort by” works fine..Clever comparator knows that valuesare ints!

Which field type - solr.SortableIntField

● q = price_field:[1 TO 10]○ "numFound": 10

● “Sortable”, in fact, refer to the notion of making the numbers have correctly sorted order. It’s not about “sort by” actually!

● Processed and compared as strings!!!tricky string encoding:NumberUtils.int2sortableStr(...)

● Deprecated and will be removed in 5.X● What should i use then?

Which field type - solr.TrieIntField

● q = price_field:[1 TO 10]○ "numFound": 10

● Recommended as replacement for IntField and SortableIntField in javadoc

● Default for primitive fields in reference schema

● Said to be fast for range queries (actually depends on precision step)

● Tricky and, btw wtf is precision step?

Agenda:

● What is RangeQuery

● Which field type to use for Numerics

● Range stuff under the hood (run!)

● NumericRangeQuery

● Useful links

Under the hood - Index

Under the hood - Index

NumericTokenStream is where half of magic happens! ● precision step = 1● value = 11

● Let’s see how it will be indexed!

00000000 00000000 00000000 00001011

Under the hood - Index

Field with precisionStep=1

Under the hood - Index00001011

00001010

00001000

11

10 = 5 << 1

8 = 2 << 2

shift=0

shift=1

shift=2

00001000 8 = 1 << 3shift=3

00000000 0 = 0 << 4shift=4

continue…00000000 0 = 0 << 5shift=5

Under the hood - Index

11111111 11111111 11111111 11111111

Algorithm requires to index all 32/precisionStep terms

So, for “11” we have 11, 10, 8, 8, 0, 0, 0, 0, 0….0

How much for an integer?

Under the hood - Index

Okay! We indexed 32 tokens for the field. (TermDictionary! Postings!) Where is the trick?

Stay tuned!

Under the hood - Query

Under the hood - Query

Sub-classes of FieldType could override #getRangeQuery(...) to provide their own range query implementation.

If not, then likely you will have:MultiTermQuery rangeQuery = TermRangeQuery.newStringRange(...)

TrieField overrides it. And here comes...

Agenda:

● What is RangeQuery

● Which field type to use for Numerics

● Range stuff under the hood (run!)

● NumericRangeQuery

● Useful links

Numeric Range Query (Decimal)● Decimal example, precisionStep = ten● q = price:[423 TO 642]

Numeric Range Query (Binary)● precisionStep = 1● q = price:[3 TO 12]

1 2 3 4 5 6 7 8 9 10 11 12 130

Numeric Range Query (Binary)● precisionStep = 1● q = price:[3 TO 12]

1 2 3 4 5 6 7 8 9 10 11 12 130

0 1 2 3 4 5 6

SHIFT = 1

Numeric Range Query (Binary)● precisionStep = 1● q = price:[3 TO 12]

1 2 3 4 5 6 7 8 9 10 11 12 130

0 1 2 3 4 5 6

0 1 2 3

0 1

0

0

0

...

Numeric Range Query (Binary)● precisionStep = 1● q = price:[3 TO 12]

1 2 3 4 5 6 7 8 9 10 11 12 130

0 1 2 3 4 5 6

0 1 2 3

0 1

0

Numeric Range Query (How?)

So, the questions is: How to create query for the algorithm?

Numeric Range Query (How?)Let’s come back to TrieField#getRangeQuery(...)

There are several options:● field is multiValued, hasDocValues, not indexed

○ super#getRangeQuery● field is hasDocValues, not indexed

○ new ConstantScoreQuery ( FieldCacheRangeFilter.newIntRange(...) )

● otherwise ta-da○ NumericRangeQuery.newIntRange(...)

Numeric Range Query (How?)NumericRangeQuery extends MultiTermQuery which is:

An abstract Query that matches documents containing a subset of terms provided by a

FilteredTermsEnum enumeration.

This query cannot be used directly(abstract); you must subclass it and define getTermsEnum(Terms, AttributeSource) to provide a FilteredTermsEnum

that iterates through the terms to be matched.

Numeric Range Query (How?)Let’s understand how #getTermsEnum works. Returns new NumericRangeTermsEnum(...)

The main part is: NumericUtils.splitIntRange(...)

Numeric Range Query (How?)Algorithm uses binary masks very much:

1 2 30

0 1

for (int shift=0; noRanges(); shift += precisionStep): diff = 1L << (shift + precisionStep); mask = ((1L << precisionStep) - 1L) << shift;

Diff is distance between upper level neighborsMask is to check if currentLevel node has nodes lower or upper. (1,3 hasLower, 0,2 hasUpper)

diff=2

Numeric Range Query (How?)hasLower = (minBound & mask) != 0L;hasUpper = (maxBound & mask) != mask;

if (hasLower)addRange(builder, valSize, minBound, minBound | mask, shift);

if (hasUpper)addRange(builder, valSize, maxBound & ~mask, maxBound, shift);

Numeric Range Query (How?)hasLower = (minBound & mask) != 0L;hasUpper = (maxBound & mask) != mask;

nextMinBound = (hasLower ? (minBound + diff) : minBound) & ~mask;

nextMaxBound = (hasUpper ? (maxBound - diff) : maxBound) & ~mask;

Numeric Range Query (How?)// If we are in the lowest precision or the next precision is not available.

addRange(builder, valSize, minBound, maxBound, shift);

// exit the split recursion loop (FOR)

Numeric Range Query (How?)● shift = 0● diff = 0b00000010 = 2● mask = 0b00000001 = 1● hasLower = (3 & 1 != 0)? = true● hasUpper = (12 & 1 != 1)? = true

○ addRange 3..(3 | 1) = 3..3○ addRange 12..(12 & ~1) = 12..12

● nextMin = (3 + 2) & ~1 = 4● nextMax = (12 - 2) & ~1 = 10

1 2 3 4 5 6 7 8 9 10 11 12 130

Numeric Range Query (How?)

1 2 3 4 5 6 7 8 9 10 11 12 130

0 1 2 3 4 5 6

● min:4; max:10● shift = 1● diff = 0b00000100 = 4● mask = 0b00000010 = 2● hasLower = (4 & 2 != 0) ? = false● hasUpper = (10 & 2 != 2) ? = false● nextMin = min● nextMax = max

Numeric Range Query (How?)

1 2 3 4 5 6 7 8 9 10 11 12 130

0 1 2 3 4 5 6

0 1 2 3

● min:4; max:10● shift = 2● diff = 0b00001000 = 8● mask = 0b00000100 = 4● hasLower = (4 & 4 != 0) ? = true● hasUpper = (10 & 4 != 4) ? = true● nextMin = (4 + 8) & ~4 = 8 => min > max END● nextMax = (10 - 8) & ~4 = 0 => range 1..2 shift =

2

Numeric Range Query (How?)TestNumericUtils#testSplitIntRange

assertIntRangeSplit(lower, upper, precisionStep, expectBounds, shifts)assertIntRangeSplit(3, 12, 1, true, Arrays.asList( -2147483645,-2147483645, // 3,3 -2147483636,-2147483636, // 12,12 536870913, 536870914), // 1, 2 for shift == 2 Arrays.asList(0, 0, 2)); // Crappy unsigned int conversions are done in the asserts

Numeric Range Query (How?)

So, NumericTermsEnum generates and remembers all ranges to match.

Numeric Range Query (How?)Basically TermsEnum is an Iterator to seek or step through terms in some order.In our case order is:

Then (shift = 1):

Then (shift = 2)

...

1 2 3 4 5 6 7 8 9 10 11 12 130

0 1 2 3 4 5 6

0 1 2 3

Numeric Range Query (How?)Actually we have FilteredTermsEnum:

1. Only red terms are accepted by our enumerator2. If term is not accepted we advance:

FilteredTermsEnum#nextSeekTerm(currentTerm)TermsEnum#seekCeil(termToSeek)

Seek term depends on currentTerm and generated ranges.

Numeric Range Query (How?)Ok, now we have TermsEnum for MiltiTermQuery and enum is able to seek through only those terms which match appropriate sub ranges.

The question is how to convert TermsEnum to Query!?

Numeric Range Query (How?)The last trick is query#rewrite() method of MultiTermQuery (rewrite is always called on query before performing search):

public final Query rewrite(IndexReader reader) { return rewriteMethod.rewrite(reader, this); }

Oh, “rewriteMethod” how interesting… It defines how the query is rewritten.

Numeric Range Query (How?)There are plenty of different rewrite methods, but most interesting for us are:●

CONSTANT_SCORE_*○ BOOLEAN_QUERY_REWRITE○ FILTER_REWRITE○ AUTO_REWRITE_DEFAULT

Numeric Range Query (How?)

BOOLEAN_QUERY_REWRITE

1. Collect terms (TermCollector) by using #getTermsEnum(...)

2. For each term create TermQuery3. return BooleanQuery with all TermQuery as leafs

Numeric Range Query (How?)

FILTER_REWRITE

1. Get termsEnum by using #getTermsEnum(...)2. Create FixedBitSet3. Get DocsEnum for each term4. Iterate over docs and bitSet.set(docid);5. return ConstantScoreQuery over filter (bitSet)

Numeric Range Query (How?)

AUTO_REWRITE_DEFAULT

If the number of documents to be visited in the postings exceeds some percentage of the maxDoc()

for the index then FILTER_REWRITE is used, otherwise BOOLEAN_REWRITE is used.

Agenda:

● ..

● I promised. Precision Step!

● ...

Precision stepSo, what is precision step and how it affects performance?

● Defines how much terms to index for each value○ Lower step values mean more precisions and

consequently more terms in index○ indexedTermsPerValue = bitsPerVal / pStep○ Lower precision terms are non unique, so term

dictionary doesn’t grow much, however postings file does

Precision stepSo, what is precision step and how it affects performance?

● ...○ Smaller precision step means less number of

terms to match, which optimizes query speed○ But more terms to seek in index○ You can index with a lower precision step value

and test search speed using a multiple of the original step value.

○ Ideal step is found by testing only

Precision step (Results)According to NumericRangeQuery javadoc:

● Opteron64 machine, Java 1.5, 8 bit precision step● 500k docs index● TermRangeQuery in BooleanRewriteMode took

about 30-40 seconds● TermRangeQuery in FilterRewriteMode took

about 5 seconds● NumericRangeQuery took < 100ms

Agenda:

● What is RangeQuery

● Which field type to use for Numerics

● Range stuff under the hood (run!)

● NumericRangeQuery

● Useful links

Useful links● http://searchhub.org/2009/05/13/exploring-

lucene-and-solrs-trierange-capabilities/● http://www.panfmp.org/● http://epic.awi.de/17813/1/Sch2007br.pdf● http://lucene.apache.

org/core/4_3_1/core/org/apache/lucene/search/NumericRangeQuery.html

● http://en.wikipedia.org/wiki/Range_tree

● me http://plus.google.com/+VadimKirilchukkirilchukvadim@gmail.com