OSSCON: Big Search 4 Big Data

Big Search w/ Big Data Principles

Basis Technology Open Source Search 2012 Eric Pugh | [email protected] | @dep4b

Tuesday, October 2, 2012

mailto:[email protected]


What is Big Search?Tuesday, October 2, 2012

http://www.flickr.com/photos/michaelgoodin/3256925948/



Who am I?

• Principal of OpenSource Connections - Solr/Lucene Search Consultancy

• Member of Apache Software Foundation

• SOLR-284 UpdateRichDocuments (July 07)

• Fascinated by the art of software development


CO-AUTHOR

2nd edition!


Telling some stories

• Prototyping

• Application Development

• Maintaining Your Big Search Indexes

war^


Not an intro to SolrCloud!

• Great tutorials given by Tomás Fernández Löbbe from LucidWorks yesterday!


Background for Client X’s Project

• Big Data is any data set that is primarily at rest due to the difficulty of working with it.

• 100’s of millions of documents to search

• Limited selection of tools available.

• Aggressive timeline.

• All the data must be searched per query.

• On Solr 3.x line



• Prototyping




Boy meets Girl Story

Metadata

Content Files

IngestPipeline

SolrSolrSolrSolr


Bash Rocks


Bash Rocks

• Remote Solr stop/start scripts

• Remote Indexer stop/start scripts

• Performance Monitoring

• Content Extraction scripts (+Java)

• Ingestor Scripts (+Java)

• Artifact Deployment (CM)


Make it easy to change approach


Make it easy to change sharding

public void run(Map options, List<SolrInputDocument> docs) throws InstantiationException, IllegalAccessException, ClassNotFoundException { IndexStrategy indexStrategy = (IndexStrategy) Class.forName( "com.o19s.solr.ModShardIndexStrategy").newInstance(); indexStrategy.configure(options); for (SolrInputDocument doc:docs){ indexStrategy.addDocument(doc); } }


Separate JVM from Solr Cores

• Step 1: Fire up empty Solr’s on all the servers (nohup &).

• Step 2: Verify they started cleanly.

• Step 3: Create Cores (curl http://search1.o19s.com:8983/solr/admin?action=create&name=run2)

• Step 4: Create a “aggregator” core, passing in urls of Cores. (&property.shards=)


http://search1.o19s.com:8983/solr/admin?action=create&name=collection






Go Wide Quickly


shard1shard1shard1shard1 :8983



search1.o19s.com



search1.o19s.com


search2.o19s.com


search3.o19s.com


Simple Pipeline

• Simple pipeline

• mv is atomic


Don’t Move Files

• SCP across machines is slow/error prone

• NFS share, single point of failure.

• Clustered file system like GFS (Global File System) can have “fencing” issues

• HDFS shines here.

• ZooKeeper shines here.


Can you test your changes?


JVM tuning is black art-verbose:gc-XX:+PrintGCDetails-server-Xmx8G-Xms8G-XX:MaxPermSize=256m-XX:PermSize=256m-XX:+AggressiveHeap-XX:+DisableExplicitGC-XX:ParallelGCThreads=16-XX:+UseParallelOldGC



Run, don’t Walk



• Prototyping

•Application Development



Using Solr as key/value store

Metadata

Content Files

IngestPipeline

SolrSolrSolrSolr

Solr Key/Value Cache


• thousands of queries per second without real time get.

• how fast with real time get?

http://localhost:8983/solr/run2_enrichment/select?q=id:DOC45242&fl=entities,html

http://localhost:8983/solr/run2_enrichment/get?id=DOC45242&fl=entities,html

Using Solr as key/value store


http://localhost:8983/solr/us_patent_grant/update/csv?skipLines=1&fieldnames=abstract,main-classification,invention-title,date-publ,description,id,further-classification,date-produced,claim-text&commit=true&optimize=true'








Push schema definition to the application

• Not “schema less”

• Just different owner of schema!

• Schema may have common set of fields like id, type, timestamp, version

• Nothing required.

q=intensity_i:[70 TO 0]&fq=TYPE:streetlamp_monitor


Don’t do expensive things in Solr

• Tika content extraction aka Solr Cell

• UpdateRequestProcessorChain


Don’t do expensive things in Solr

• Tika content extraction aka Solr Cell

• UpdateRequestProcessorChain


Beware JavaBin

Metadata

Content Files

IngestPipeline

SolrSolrSolrSolr



Beware JavaBin

Metadata

Content Files

IngestPipeline

SolrSolrSolrSolr


Solr 3.4


Beware JavaBin

Metadata

Content Files

IngestPipeline

SolrSolrSolrSolr


Solr 3.4

Solr 4


Beware JavaBin

Metadata

Content Files

IngestPipeline

SolrSolrSolrSolr


Solr 3.4

Solr 4

Which SolrJ version do I

use?


No JavaBin

• Avoid Jarmaggeddon

• Reflection? Ugh.

Give m

e

/update/avro!


Avro!

• Supports serialization of data readable from multiple languages

• It’s smart XML, w/o the XML!

• Handles forward and reverse versions of an object

• Compact and fast to read.


Avro!

Metadata

Content Files

IngestPipeline

SolrSolrSolrSolr


.avro



• Prototyping


•Maintaining Your Big Search Indexes


Upgrade Lucene Indexes Easily

• Don’t reindex!

• Try out new versions of Lucene based search engines.

David Lyle

java -cp lucene-core.jar org.apache.lucene.index.IndexUpgrader [-delete-prior-commits] [-verbose] indexDir


Indexing is Easy and Quick


CHEAP AND CHEERFUL

><


NRT versus BigData


The tension between scale and update rate

10 million 100’s of millionsBad Place


Grim ReaperTuesday, October 2, 2012

Delayed Replication<requestHandler name="/replication" class="solr.ReplicationHandler" ><lst name="slave"> <str name="masterUrl">http://localhost:8983/solr/replication</str> <str name="pollInterval">36:00:00</str></lst></requestHandler>


http://localhost:8983/solr/replication

http://localhost:8983/solr/replication

Enable/Disable

• Solr-3301


Enable/Disable

<requestHandler name="/admin/ping" class="solr.PingRequestHandler"><lst name="invariants"> <str name="q">MY HARD QUERY</str> <str name="shards">http://search1.o19s.com:8983/solr/run2_1,http://search1.o19s.com:8983/solr/run2_2,http://search1.o19s.com:8983/solr/run2_2</lst><lst name="defaults"> <str name="echoParams">all</str></lst><str name="healthcheckFile">server-enabled.txt</str></requestHandler>


http://search1.o19s.com:8983/solr/run2_1








Provisioning

• Chef/Puppet

• ZooKeeper

• Have you versioned everything to build an index over again?


TRADITIONAL ENVIRONMENT


POOLED ENVIRONMENTthink Cloud!


Do I need Failover?

• Can I build quickly?

• Do I have a reliable cluster of servers?

• Am I spread across data centers?

• Is sooo 90’s....



• Prototyping




One more thought...


Measuring the impact of our algorithms

changes is just getting harder with Big Data.


Project SolrPanlTuesday, October 2, 2012

Thank you!

Questions?

• [email protected]

• @dep4b

• www.opensourceconnections.com




http://www.opensourceconnections.com

http://www.opensourceconnections.com