Invisible Web or Deep Web


The "visible web" is what you see in the results pages from general web search engines. It's also what you see in almost all subject directories. The "invisible web" is what you cannot retrieve ("see") in the search results and other links contained in these types of tools.

The Invisible Web consists of hundreds of millions of web pages that are not indexed by the traditional search engines. The Invisible Web makes up a significant proportion of the web with an estimated size of between half and two thirds of the total. Typical content of invisible web pages include interactive tools, archived material, statistical sources, news sources and a whole variety of full text databases such as dictionaries, encyclopedias and online books.  Invisible or Deep web be broadly classified as:

Searchable Databases - Most of the invisible web is made up of the contents of thousands of specialized searchable databases that you can search via the Web. The search results from many of these databases are delivered to you in web pages that are just for your search. Such pages very often are not stored anywhere: it is easier and cheaper to dynamically generate the answer page for each query than to store all the possible pages containing all the possible answers to all the possible queries people could make to the database. Search engines cannot find or create these pages.

Excluded Pages - There are some types of pages that search engine companies exclude by policy. There is no technical reason they could not include them if they wanted. It's a matter of selecting what and what not to include in databases that are already huge, expensive to operate, and low revenue producers.

How to Find the Invisible Web -

  1. Simply think "databases" and keep your eyes open? You can find searchable databases and other invisible web stuff in the course of routine searching in most general web directories.

  2. Use Google and other search engines to locate searchable databases by searching a subject term and the word "database". If the database uses the word database in its own pages, you are likely to find it in Google. The word "database" is also useful in searching a topic in Yahoo!, because Yahoo! used the term to describe searchable databases in its listings.


Below is a listing of search tools and subject directories that can help you locate information on the web that is usually not found by using standard web search engines.

Argus Clearinghouse -  Librarians and subject specialists rate web pages by subject area.

BUBL - UK Subject tree of scholarly resources on the web.

CompletePlanet - Finds databases on the web, not web content.

Digital Librarian - A librarian's choice of the best of the web.

Direct Search - http://www.freepintcom/gary/direct.htm A growing compilation of links to the search interfaces of resources.

Find - Free online article-search service from more than 300 reputable magazines and journals.

FirstGov - U.S Government search engine for federal and state government information. -  Searches a selective collection of 3000 specialized databases and topical guides.

Google Uncle Sam -  Searches U.S. government sites on the web.

GPO Access -  This is the website of the U.S. government printing office.

Infomine – Scholarly Internet Resource Collection -  Provides search from more than 40,000 academic resources.

Ingenta-  Ingenta provides with access to comprehensive collection of academic and professional content available online.

Internet Public Library -  Good starting point for finding reference works, subject guides and specialized databases.

The Invisible Web - A comprehensive directory to thousands of specialized databases.

Librarian’s Index to the Internet - A well-organized point of access for reliable, trustworthy, librarian-selected Internet resource

LibrarySpot - Links to quality reference resources and links to more than 2500 libraries worldwide. - Indexes and links to magazine articles available for free on the Web.

Open Directory -  Uses thousands of volunteers to create listings and create an organized Internet Directory owned by Netscape.

Scirus - Comprehensive science-specific search engine launched by Elsevier Science. -  Search engine created by U.S. government agencies on scientific research.

Scout Report - Provides way to keep up-to-date with new search tools and resources.

Search Adobe PDF Online -  Search through more summaries of Adobe® Portable Document Format (PDF) files on the Web.

World Wide Web Virtual Library -  Annotated lists of sites compiled by experts in individual subject areas.

Naina Pandita, NIC, New Delhi