The Invisible Web

DefinitionSearching

The Invisible Web

Also called: deep content hidden internet dark matter

The Invisible Web

The vast number of pages that search engines cannot or will not index

Restricted: login, password (such as intranets, databases; private, proprietary)

Sites not linked from anywhere (undiscovered) Sites that use a robots.txt file to keep files off limits from spiders Unsearchable or un-indexable file formats Non-static - searchable databases that only produce results

dynamically in response to a specific search request (such as CGI, ASP, CFM)

Real-time data – changes rapidly – too “fresh” Sites that are too “deep”

The Invisible Web

Search engines often avoid indexing web pages that are delivered dynamically, such as via database programs:

Often, the search engine may not like the URL used in order to retrieve the document. Many dynamic delivery mechanisms make use of the ? symbol.

For example, a page may be found this way:http://www.website.com/cgi-bin/getpage.cgi?name=sitemap Most search engines will not read past the ? in that URL.

The Invisible Web

Invisible Web sources tend to be: More current More comprehensive Searchable (however, not by SE’s) More specific/targeted Deeper breadth Often better quality

The Invisible Web

Top types of “invisible” information News RSS Blogs Public company filings, stock prices Customized maps and directions Clinical trials Telephone numbers and addresses, postal codes Definitions Job postings Grant information Statistics Weather Museum, gallery, and library holdings

Finding the “Dark Matter”

Search Engines Specialized Search Engines Directories Vortals

Traditional Search Engines

Traditional Search Engines incorporation of “Invisible” Databases

Weather Maps Phone directories Catalogs Stock prices

Traditional Search Engines

Unless specially, programmed, though, spiders can’t find all the valuable resources available

Specialized Search Engines

Search deeper into sites: Go beyond top page, or homepage Choose sources to spider—topical sites

only “Smart” ranking and indexing based on

knowledge of the specific subject

Specialized Search Engines

There are hundreds of specialized search engines for almost every topic-

Search Engine Guide Specialty Search Engines

Directories

Collections of pre-screened web-sites into categories based on a controlled ontology

Ontology: classification of human knowledge into topics, similar to traditional library catalogs

Directories

Closed Model: paid editors; quality control (LookSmart, Yahoo)

Open Model: volunteer editors; (Open Directory Project, Google)

Directories

Easier access to relevant results Faster Access to materials not always indexed by

search engines—content in databases or file types not searched by spiders

Directories

Issues with directories: Inherently small Unseen editorial policies

May charge for listingLopsided coverage

Timeliness--Harder to keep updated

SearchSearch

Vortals

Vortals: vertical-portal. Instead of being a horizontal, all-inclusive entry point into the Web, they are vertical, specialized entry points.

Comprehensive sites focusing on gathering and providing links to the best resources in a specific topic.

Usually are combined subject-specific search engines and subject-specific directories

Also called “focused crawlers”; metasites; guru; authority; industry guide; subject directory site

Vortals

Advantages – best of directories and subject specific search engines

More up-to-date - crawl subject specific pages more often

Deeper crawl - gets more of the content on each server

More precision, less recall

Searching the Invisible Web

How do you find these sites? Use directories known directories to find

invisible web searching and browsing tools:Librarians’ Index to the InternetOpen DirectoryGoogle DirectoryTeoma works well, too.

Searching the Invisible Web

Rethink your search: Think key terms specific details – macro vs. micro Example you want to find the melting point of hydrogen

peroxide. On the general web, you’d put in the key words melting, point, and “hydrogen peroxide” On the invisible web, you look for chemical databases, which included melting points as one feature of the database, once in the database, then you’d search for hydrogen peroxide

Searching the Invisible Web

Remember some concepts are assumed Do not use the subject a search term Example: If you are looking for information

on gender inequity in math education, exclude terms like education from your search in AskERIC, an education specific search tool

Mining the Invisible Web

Tips: Certain kinds of sites can prove to be clearinghouses of information:

Government - statistics of all kinds Professional organizations - archives of relevant

research and statistics Media sites (TV and Radio) – transcripts and

speeches College and university professor sites – lectures

and personal publications

Mining the Invisible Web

Look for library guides and commercial portals for more guidance in finding the hidden, valuable content available for free on the Web (more on this in the next lesson):

My Ready Reference on the Web Resource