The Invisible Web

By Jake SheehanNick Saltzman

What is the invisible web? All information that cannot be indexed using

general web search engines

Also known as the deep internet, deepnet, or the hidden web

Search Engines Programs that search documents for

specified keywords or phrases and returns a list of documents where the keywords were

found

How Search Engines Work Uses a spider program to fetch as many

webpages as possible

A program called an indexer then reads these webpages and creates an index,

storing the URL and important content of webpage

Each search engine has its own ranking algorithm that returns results based on their relevance to the user’s specified keywords or

phrases

Spider Programs Starts by searching heavily used servers and

the most popular web pages

Visits all the links found in the webpages it reads and sends them to the search engine’s indexer in a process called web crawling,

which continues automatically

To be discovered, a webpage must be static and linked to other pages

All pages that are indexed by a search engine’s spider programs are known as the

visible web

The Invisible Web Everything not found by a search engine’s

spider program is part of the invisible web

Most webpages on the internet are not indexed by search engines

Some experts estimate that as much as 75% of the entire internet is invisible web content

Content Found On The Invisible Web

Webpages with no links on them, called disconnected pages

Password protected webpages Webpages generated from databases

Dynamically generated webpages Real-time content

Webpages that require a registration form to access

Webpages with non-html text, or any coding that a spider program can not understand

Advantages Of Invisible Web Content

Specialized content focus – large amounts of information focused on an exact subject

Contains information that might not be available on the visible web

Allows a user to find a precise answer to a specific question

Allows a user to find webpages from a specific date or time

So what is actually on the Deep Web, and how do we get there?

Deep Web (layer two)- Content that is only available through

specialized anonymizing software (Tor). This includes anonymity networks and darknets (anonymous P2P). This are is a sort of safe- haven for speech and content which has been banned from the surface web.

- examples: Anonymous chat/imageboards. CP, drugs, illeagal transactions, weapons, DRM removal software, pirated software/media/books, black markets

Accessing the Deep Web

.onion: -onion is a domain host suffix designating an anonymous hidden service reachable via the Tor network.

-The purpose of using such a system is to make both the information provider and the person accessing the information more difficult to trace, whether by one another, by an intermediate network host, or by an outsider.

-.onion adresses are 16-character non-mneumonic hashes, compromised of alphabetic and numeric strings.

-The "onion" name refers to onion routing, the technique used by Tor to achieve a degree of anonymity.

Accessing the Deep Web

Tor:-” The Tor software protects you by bouncing your

communications around a distributed network of relays run by volunteers all around the world: it prevents somebody watching your Internet connection from learning what sites you visit, it prevents the sites you visit from learning your physical location, and it lets you access sites which are blocked.” – www.torproject.org

-Tor is a network that supports onion routing; a way to help make your traffic anonymous. Because the Deep Web is compromised of information that doesn’t show up on search engines, or has no domain name registry, you must know exactly where you are going to get there.

The first step is to download, install, and run the Tor Bundle. This will bring up your new anonymous browser (through Firefox) in order to get started.

Guided Journey Through the Deep Web

The deep web is vast, as we covered before, and most is private. Here we will go through some examples of websites on the Deep Web, how online currency works, and the various resources the Deep Web encompasses (not all entirely legal)

This content can be disturbing. Obviously with great resources such as free library databases, and banned books, comes information that is considered scat. Being a safe-haven from the censorship of the surface web, there is a lot of CP, arms/drug trade, and other illegal information on the Deep Web. It’s up to you as an individual to steer clear of what you don’t want to see.

In addition, one must also be skeptical. There are even sites dedicated to contract killing, but one can’t be sure that these websites are even serious.

The next step is to access the Hidden Wiki, which most people consider the home page of the Deep Web. Here you can begin your journey and discover many different types of sites and networks, ranging from tame to very illegal. There are black market type sites, hacking information sites, huge file databases, political advocacy sites, and even sites to hire people to engage in illegal activity.

One of the other “main pages” of the Deep Web is TorDir. TorDir is a site that uses a special crawler for the Deep Web, so that it may act similar to Google in that it categorizes web sites, and allows you to view many different and obscure .onion sites that fall into each category.

Through the Hidden Wiki you can find pages like this, which are semi-organized lists of different .onion sites. Many different sites are listed and separated based on function, such as buying/selling/trading, communication, hacking, or intel exchange.

Currency of the Deep WebBitcoins:

-an electronic cash system using peer-to-peer networking, digital signatures and cryptographic proof to enable irreversible payments between parties without relying on trust. Payments are issued and transferred by the Bitcoin network.

-Participants begin using Bitcoin by first acquiring a Bitcoin wallet and one or more Bitcoin addresses. Bitcoin addresses are used for receiving Bitcoins, similar to how e-mail addresses are used for receiving e-mail.

-In April 2012, 1 BTC traded at around $4.80. Taking into account the total number of Bitcoins in circulation, the market capitalization of the Bitcoin network stands at over 40 million USD

This is an example of one of the many online black markets. This one did not require a paid membership. There are many different ways to spend bit coins, such as on apparel, money transfers, drugs, books, and even digital goods.

This is an example of someone’s own personal page, with content picked out due to it’s controversial nature. Here you can find anything from banned readings, to trefelling. There is content on making one’s own explosives, and many different military type documents and guidebooks.

There is a huge portion of the Deep Web that is not malicious at all. This is a huge database of banned, black listed, and forbidden books which you can read and download. Books have been burned in the past and banned for their controversial subject matter, and this is a way for history to be preserved.

Another scholarly database, this .onion site is a compilation of financial, business, real estate, and even marketing tools and texts free to anyone who wants to download them. The Deep Web is a great archive for educational material.

The Deep Web is full of malicious content. On this Russian-based community hacking website, one can purchase credit card information, PayPal account information, bank accounts, and even the service of DDoSing a website. This is when the website is overloaded with information sent to it, and eventually is forced to shut down.

Dangers of the Deep Web

If you decide to look into the Deep Web on your own volition, make sure to be careful.

-Have anti-virus protection.

-Be intelligent about what links you click. The Deep Web is a haven for Phishers.

-If you don’t want to see any disturbing images or content, simply browse as text-only.