Embed Size (px)
Citation preview
Web &
Working of Search Engine
Presented By:
Vinay Arora
Assistant Professor
CSED, Thapar University
Web Content
� Web Content/Resource means content accessible/present on Internet.
� Visible Web – The Publicly Index able pages that have been picked up and Indexed by conventional search engines, mainly consist of static HTML pages.
� Invisible Web/Deep Web/Hidden Web - Information that cannot be Indexed/Seen by the Crawlers or Spiders of conventional Search Engines.
� Types of Invisible Web
Invisible Web
Visible Web
OpaquePrivate
Proprietary
Truly Invisible Web
TYPES of Invisible Web & Reasons of being Invisible
� Truly Invisible Web is not accessible for search engines mainly because of technical reasons Dynamically generated pages, Pages with pdf, exe, swf format.
� Proprietary Web Databases which are mainly fee based and are provided by
Information Providers. These Databases provide user with search facility however, their contents are not searchable through the search engines.
� Private Web Technically Indexable , but have purposely been excluded from search engines using Password Protected Pages, Robot.txt, NoIndex META Tag.
� Opaque Web Disconnected URL.
Size Of Invisible Web is approx.500 times larger than Visible Web.
Crawling & Indexing
A Search Engine operates, in the
Following order:
1. Web Crawling.
2. Indexing.
3. Searching.
Query Processing/Searching
Making Invisible Web Visible
� Register Website with Search Engine
Making Invisible Web Visible
� Sitemap.xml - Sitemaps are an easy way for webmasters to inform search
engines about pages on their sites that are available for crawling. In its simplest form, a Sitemap is an XML file that lists URLs for a site along with additional metadata about each URL.
Making Invisible Web Visible
� Making Entries into Robot.txt file for allowing the Robots to Crawl and Changing
META Content.
Making Invisible Web Visible
� Providing links of the desired website from another Websites so that it can be
made accessible from other/different websites. And can be Crawled.
� Changing the Source Code of Web Crawlers – Making the crawlers efficient and
intelligent enough so that it can accept files with extension pdf, swf etc. and list/Index the entries properly.
� The content of Proprietary Web Databases are not searchable through the
search engines. They are assembled into Web pages as responses to queries submitted through the “Query Interface” of an underlying database. Because current search engines cannot effectively “Crawl” databases, such data is
believed to be “Invisible,” and thus remain largely “hidden” from users
www.orkut.comwww.gmail.comorkut
Conceptual View Of Deep Web
Conceptual View Of Deep Web
Google Advance Search
Google Advance Search
User Form Interaction
� For Form-based Search Interfaces when user is present for Input instead of
Crawler. Result will be obtained after Query execution as soon as User press Submit button after filling the required fields present in the Form.
We have to make this Visible.We want Response Page to be
listed in Search Engine.
Crawler Form Interaction & Steps for Hidden Web Crawler
� Crawler at desired URL.
� Form Analysis for Internal Form Representation.
� Matching with the entries present in Task Specific Database.
� Automatic FORM Processing and Submission.
� Response Page from the Server.
� Response Analysis of that Page.
� Putting the results in the Repository.
References
� The Deep Web: Surfacing Hidden Value. http://www.completeplanet.com/Tutorials/DeepWeb/.
� Paper: Crawling the Hidden web Hector Garcia CSE Department Stanford University, USA
� http://www.invisible-web.net
� All About Invisible Web : Natalia Arroyo, Internet Lab, CINDOC – CSIC
� Accessing the Deep Web: A Survey , Bin He, Mitesh Patel, Zhen Zhang, Kevin Chen-Chuan Chang, Computer Science Department, University of Illinois at
Urbana-Champaign.
� Towards a Model of User oriented Aspects of the Invisible Web, Yazdan
Mansourian, Department of Information Studies , The University of Sheffield
