Anatomy of a search engine

• Design criteria of a search engine• Architecture• Data structures

Step-1: Crawling the web

Google has a fast distributed crawling systemEach crawler keeps roughly 300 connection open at onceGoogle can crawl over 100 web pages per second using four crawlers at peak speeds (roughly 600K per second of data)Each crawler maintains a its own DNS cacheThe crawler uses asynchronous IO and a number of queues

Ref: http://www.elibrary.icrisat.org/Google%20Search/how%20google%20works.htm

Step2: Indexing the Web (1)• Parsing

Any parser which is designed to run on the entire Web must handle a huge array of possible errors. These range from typos in HTML tags to kilobytes of zeros in the middle of a tag, non-ASCII characters, HTML tags nested hundreds deep, and a great variety of other errors that challenge anyone's imagination to come up with equally creative ones.

Use flex to generate a lexical analyzer for maximum speed

URL Server Store ServerCrawlerCrawlerCrawler

RepositoryIndexer

Barrels

IndexerIndexerSorter

Step2: Indexing the Web (2)Indexing Documents into Barrels

After each document is parsed, it is encoded into a number of barrels.

Every word is converted into a wordID by using an in-memory hash table -- the lexicon.

New additions to the lexicon hash table are logged to a file.

The words in the current document are translated into hit lists.

The words are written into the forward barrels.

For parallelization, indexer writes a log to a file, instead of sharing the lexicon

Sorting

Takes each of the forward barrels.

Sorts it by wordID to produce an inverted barrel.

Parallelize the sorting phase.

Subdivides the barrels into baskets to load into main memory because the barrels don’t fit into memory.

Sorts baskets and writes its contents into the inverted barrel

Searching (pseudo code)

1. Parse the query2. Convert words into wordIDs3. Seek to the start of the doclist in the short barrel for every word4. Scan through the doclists until there is a document that matches all the

search terms5. Compute the rank of that document for the query6. If we are in the short barrels and at the end of any doclist, seek to the

start of the doclist in the full barrel for every word and go to step 47. If we are not at the end of any doclist go to step 48. Sort the documents that have matched by rank and return the top k

Figure 4. Google Query Evaluation

SearchingThe Ranking System

Every hit-list includes position, font and capitalization information.

Factor in hits from anchor text and the PageRank of the document.

Ranking function so that no particular factor can have too much influence

For a single word search

o In order to rank a document, Google looks at that document’s hit list for a single word query and computes an IR score combined with PageRank

For a multi-word search

o Hits occurring close together in a document are weighted higher than hits occurring far apart

Use of Feedback

Google has a user feedback mechanism because figuring out the right values for many parameters is very difficult.

When the ranking function is modified, this mechanism gives developers some idea of how a change in the ranking function affects the search results

Page Rank

Page Rank: Page Rank of any node shows: “how many important or popular nodes vote for the given node”. This is a collective intelligence of nodes in graph.

For given vertex/node 'iV , let 'iVIN be the set of vertices/nodes that point to it

(predecessors), and let 'iVOUT be the set of vertices that vertex 'iV points to (successors).

The score of vertex 'iV can be defined as “Page et al. (1998)”:

' '

'1'

iVINj j

ji VOUT

VS

NVS

(1)

Where:

'iVS Rank of vertex iV .

'jVS =rank of vertex 'jV , from which incoming link comes to word / vertex 'iV .

N Count of number of words/vertex in word graph of sentences.

damping factor ( “0.85” is used in “Page et al. (1998)”).

Google architecture

• URL server sends list of URLs to be fetched to crawlers

• StoreServer compresses and stores pages

• Indexer extracts words, their pos., size, capital.

• Anchors cont.links and their text

• Sorter generates inverted index

• Searcher uses Lexicon, II, and PR

Major data Structure

Major Data Structures: Google's data structures are optimized so that a large document collection can be crawled, indexed, and searched with little cost. Google is designed to avoid disk seeks whenever possible, and this has had a considerable influence on the design of the data structures.

BigFiles: BigFiles are virtual files spanning multiple file systems and are addressable by 64 bit integers.

Repository: The repository contains the full HTML of every web page. Each page is compressed using zlib. The repository requires no other data structures to be used in order to access it. This helps with data consistency and makes development much easier;

Document Index: The document index keeps information about each document. It is a fixed width ISAM (Index sequential access mode) index, ordered by docID. The information stored in each entry includes the current document status, a pointer into the repository, a document checksum, and various statistics.

Additionally, there is a file which is used to convert URLs into docIDs. It is a list of URL checksums with their corresponding docIDs and is sorted by checksum. In order to find the docID of a particular URL, the URL's checksum is computed and a binary search is performed on the checksums file to find its docID.

Major data Structure

Lexicon: The current lexicon contains 14 million words (though some rare words were not added to the lexicon). It is implemented in two parts -- a list of the words (concatenated together but separated by nulls) and a hash table of pointers.

Hit Lists: A hit list corresponds to a list of occurrences of a particular word in a particular document including position, font, and capitalization information. Hit lists account for most of the space used in both the forward and the inverted indices.

Forward Index: The forward index is actually already partially sorted. It is stored in a number of barrels (we used 64). Each barrel holds a range of wordID's. If a document contains words that fall into a particular barrel, the docID is recorded into the barrel, followed by a list of wordID's with hitlists which correspond to those words.

Inverted Index: The inverted index consists of the same barrels as the forward index, except that they have been processed by the sorter. For every valid wordID, the lexicon contains a pointer into the barrel that wordID falls into. It points to a doclist of docID's together with their corresponding hit lists. This doclist represents all the occurrences of that word in all documents.

Reference

• [Cho 98] Junghoo Cho, Hector Garcia-Molina, Lawrence Page. Efficient Crawling Through URL Ordering. Seventh International Web Conference (WWW 98). Brisbane, Australia, April 14-18, 1998.

• [Page 98] Lawrence Page, Sergey Brin, Rajeev Motwani, Terry Winograd. The PageRank Citation Ranking: Bringing Order to the Web.

• [Chakrabarti 98] S.Chakrabarti, B.Dom, D.Gibson, J.Kleinberg, P. Raghavan and S. Rajagopalan. Automatic Resource Compilation by Analyzing Hyperlink Structure and Associated Text. Seventh International Web Conference (WWW 98). Brisbane, Australia, April 14-18, 1998.

• [Gravano 94] Luis Gravano, Hector Garcia-Molina, and A. Tomasic. The Effectiveness of GlOSS for the Text-Database Discovery Problem. Proc. of the 1994 ACM SIGMOD International Conference On Management Of Data, 1994.

• Sergey Brin and Lawrence Page; The Anatomy of a Large-Scale Hypertextual Web Search Engine; Computer Science Department, Stanford University, Stanford, CA 94305