Embed Size (px)
Citation preview
What is the Visible (Surface) Web?
“It’s made up of HTML Web pages that the search engines have chosen to include in their
indices. It’s no more complicated than that.”
Sherman and Price.
What is the Visible (Surface) Web?
•A collection of webpages •Searchable with “search engines”•What you and I think of as the “Internet” is actually only a small portion of the Internet
What is the Visible (Surface) Web?
•High volume
•Mass appeal
•High value
•Small percentage of web content –Exception: Google books and Google Scholar
What is the Invisible Web?
•What search engines do not search•Searchable Databases
–Tens of Thousands–Accessible and searchable via the Internet–Results often dynamically generated in specific response to your request (eBay, MapQuest, etc.)
What is the Invisible Web?•Excluded Pages
–Excluded per search engine–Excluded per webpage by the owner of the site
•Typically databases–Businesses–Governments–Schools–Libraries –Associations
What is the Invisible Web?•Academic•Never been indexed or linked•Uniquely generated pages•Proprietary •Confidential•Protected by username & password•Constitutes the majority of the webpages on the Internet
•The Invisible Web is about 550 times larger than the visible web and is growing much faster•The deep Web consists of about 91,000 terabytes. •The surface Web is only about 167 terabytes1•The Library of Congress contains about 11 terabytes. •Quality content is 1,000 to 2,000 times greater than surface web•95% of the Deep Web is accessible to public (no fees or subscription required)
•based on extrapolations from a study done at University of California, Berkeley
Visible vs. Invisible Web
Opaque Web
Private Web
Proprietary Web
Pay per click
What is on the Invisible Web
• Requires payment
• Requires registration
• Dynamically generated
• Very new
• Website specifically stops spiders
Why can’t Google find it?
• Fixed, or Could be indexed, but is not
• Deemed not important enough
• Too new and therefore not linked
• Never makes max results cutoff
• No one ever linked or submitted URL
Opaque Web
Private Web
• Deliberately excluded– Password– Special coding in website stops spiders
• Only for select individuals– Employees– Students – Researchers
Proprietary Web• Protected
– Password – Registration (N.Y. Times, eBay, banks, etc.) – Terms of Use
• Anyone can access if you – Pay – Register– Agree to terms
Pay per click
Search Engine Marketing toolsEx: overture.com, FindWhat.com
When do I use ….
• Portal or Directory?
• Search Engine?
• Invisible Web?
Portal or Directory
• You have a general topic• You know little about the subject• You do not know keywords • You want someone or something to
have sorted out the junk• You need an exploratory overview
Search Engine
• You are looking for something specific• You have keywords• You are pretty sure the information is
– advertised or – otherwise generally disseminated
Tips for search engines
• Use a toolbar• Determine the key words/phrases
most likely to be in your document and nowhere else
• Learn and use Boolean Operators• Scan results • Question the results
Invisible Web• You are pretty sure the information is in a
specific database • Need something authoritative• Speed• The information is dynamically generated• You are familiar with the database
– Search techniques– Protocols– Access requirements
Searching the Invisible Web• Directories – subject guide compiled by
human editors
• Specialized Search Engines– http://library.albany.edu/internet/choose.html
• Special Databases ( Library of Congress,Library of Congress
http://catalog.loc.gov
LookSmart’s Find Articles (over 900 publicationshttp://www.findarticles.com
National Science Digital Libraryhttp://www.nsdl.org
Singing Fish – audio and videohttp://www.singingfish.com
Special Databases
• Library of Congress– http://catalog.loc.gov
• LookSmart’s Find Articles (over 900 publications)– http://www.findarticles.com
• National Science Digital Library– http://www.nsdl.org
• Singing Fish – audio and video– http://www.singingfish.com
Types of Databases
Information stored in tables (Access, Oracle, SQL Server, DB2) and accessible only by query.
Examples: • Phone books, People finders, • Patents, laws• Items for sale in a Web store or Web-based auctions • Digital exhibits• Multimedia and graphical files• Stock and bond prices
Types of Hidden Info
• Pages in searchable databases: medical (WebMD.com), patent, scientific, legal (Lexis and Westlaw), reference
• Pages requiring login or registration: Social Sites, New
York Times, web based applications, calendars, Google Docs, etc. • Government publications or databases: ERIC,
usa.gov • Online databases: Gale Research• PDF files, audio, video, any new format
More hidden stuff
• Dictionaries and thesauri• Sites that require forms to be filled out (ex:
travel direction, job hunting)• Product catalogs and library catalogs• Newspaper and magazine archives• Dynamic web pages (ex: airline flight
checkers, mapquest)• Interactive tools (ex: calculators &
measurement converters)
Access to invisible web is improving …
Google Books http://books.google.com/
Google Scholar http://scholar.google.co.il/
Maybe Consider …
• Specialized Databases such as Dialog, Nexis Lexis, Factiva, etc. (not cheap)
• Use an Information Professional www.aiip.org
To Conclude …
Focus and continue doing what you do best and what you have been trained for and let an Information Professional find the info you need.
He is trained to do it faster, more effectively and efficiently than you or one of your employees. (www.aiip.org)
