Combining Systems and Databases: A Search Engine Retrospective

By:Rooma RathoreRohini Prinja

Author: Eric A. Brewer

Problem Statement:

How Search Engines (SEs) should have been designed.

How to leverage the Database principles in designing Data-Intensive (DI) applications without necessarily using the same semantics.

Importance of the paper in the current context:

Search Engines have become an important part of life for billions of people. It is intriguing how SE’s manage magnanimous amount of data of the order of 3B documents and increasing every second.

Behind-the-scenes challenges of designing SEs in terms of:

Ranking Documents Ranking Query Results Availability Freshness of Data

Discusses data-intensive applications in the wake of SEs.

Finally this paper invokes thought as to how scalable these models can be as in the case of SE’s the data on the internet is increasing every second.

Contributions:

It gives numbers for various search engine parameters like: No of documents, Data stored, No of queries

etc.. etc.. Discusses the challenges of designing a

SE. What principles of DBMS can be (should

be) applied when designing DI applications like search engines: Top-Down Design Data Independence Declarative Query Language

Contributions (contd.):

Why SE’s did not use DBMS in the first place?

Why SE’s can not be implemented as DBMS in true sense: Speed: DBMS are slow Cost: DBMS are not cost-effective given the

magnitude of the data High-Availability vs. Consistency: DBMS prefer

consistency in antithesis to SE’s Update: The model of updating data in SE’s is

entirely different from databases

Contributions (contd.): New Design Uses static databases and large degree of offline

work to build and rebuild static databases. Overview of SE design:

Crawl, Index, Serve Query (read-only)

Scoring of documents and words Making a Query Plan Query Implementation

Access Methods and Physical Operators Optimize queries to maximize the through-put of the

system Providing redundancy using clustering Compression and other optimizations

Updation of data Fault Tolerance

Contributions (contd.):

SE challenges different from traditional DBMS: Personalization Logging Query rewriting Phrase queries

Validations:

Author experience on developing Inktomi search engine to come up with improved search engine design.

Author also studied the working of various modern Search Engines like Google, Alta-vista, Infoseek.

Assumptions:

Following are the assumptions that author has made while writing this paper: DI applications are essentially like SE’s and

hence should be no different when it comes to utilizing database principles.

While scoring the document, author assumes that shorter the length of the document, the higher the score it should be assigned

Updates to the systems can always happen offline.

It assumes that documents from one site are evenly distributed across the cluster nodes for load balancing.

Conclusion of the paper:

Data-intensive systems should employ the principles of databases.

Many systems are a good fit for DBMS principles (though may not use the same artifacts): Logging System Google File Systems Batch Aware distributed file system

Additional information that can be re-written/added if written today:

More emphasis and details on Logging: Companies like Google earn their moolah using

advertising (of the order of billion of dollars) How the following factors should affect

the design of a SE: Probability of Click Attacks Privacy/Copyright concerns while crawling the

web Generic Search vs. Search against a particular

domain like law or image search Comparisons of the design proposed with

one current popular search engine.

References:

http://en.wikipedia.org/wiki/ “The Anatomy of a Large-Scale

Hypertextual Web Search Engine” (1998) by Sergey Brin, Lawrence Page

Thanks!!