Invisible Web or Deep Web
The "visible web" is what you see in the results pages from general web search engines. It's also what you see in almost all subject directories. The "invisible web" is what you cannot retrieve ("see") in the search results and other links contained in these types of tools.
The Invisible Web consists of hundreds of millions of web pages that are not indexed by the traditional search engines. The Invisible Web makes up a significant proportion of the web with an estimated size of between half and two thirds of the total. Typical content of invisible web pages include interactive tools, archived material, statistical sources, news sources and a whole variety of full text databases such as dictionaries, encyclopedias and online books. Invisible or Deep web be broadly classified as:
Searchable Databases - Most of the invisible web is made up of the contents of thousands of specialized searchable databases that you can search via the Web. The search results from many of these databases are delivered to you in web pages that are just for your search. Such pages very often are not stored anywhere: it is easier and cheaper to dynamically generate the answer page for each query than to store all the possible pages containing all the possible answers to all the possible queries people could make to the database. Search engines cannot find or create these pages.
Excluded Pages - There are some types of pages that search engine companies exclude by policy. There is no technical reason they could not include them if they wanted. It's a matter of selecting what and what not to include in databases that are already huge, expensive to operate, and low revenue producers.
How to Find the Invisible Web -
Below is a listing of search tools and subject directories that can help
you locate information on the web that is usually not found by using standard web search
Argus Clearinghouse - http://www.clearinghouse.net Librarians and subject specialists rate web pages by subject area.
BUBL - http://link.bubl.ac.uk:80 UK Subject tree of scholarly resources on the web.
CompletePlanet - http://www.completeplanet.com Finds databases on the web, not web content.
Digital Librarian - http://www.digital-librarian.com A librarian's choice of the best of the web.
Direct Search - http://www.freepintcom/gary/direct.htm A growing compilation of links to the search interfaces of resources.
Find Articles.com - http://www.findarticles.com Free online article-search service from more than 300 reputable magazines and journals.
FirstGov - http://firstgov.gov U.S Government search engine for federal and state government information.
Fossick.com - http://www.fossick.com Searches a selective collection of 3000 specialized databases and topical guides.
Google Uncle Sam - http://www.google.com/unclesam Searches U.S. government sites on the web.
GPO Access - http://www.access.gpo.gov/ This is the website of the U.S. government printing office.
Infomine Scholarly Internet Resource Collection -http://infomine.ucr.edu/ Provides search from more than 40,000 academic resources.
Ingenta-http://www.ingenta.com Ingenta provides with access to comprehensive collection of academic and professional content available online.
Internet Public Library - http://www.ipl.org/ref Good starting point for finding reference works, subject guides and specialized databases.
The Invisible Web - http://www.invisibleweb.com A comprehensive directory to thousands of specialized databases.
Librarians Index to the Internet -http://lii.org A well-organized point of access for reliable, trustworthy, librarian-selected Internet resource
LibrarySpot - http://www.libraryspot.com/ Links to quality reference resources and links to more than 2500 libraries worldwide.
MagPortal.com -http://magportal.com Indexes and links to magazine articles available for free on the Web.
Open Directory - http://dmoz.org Uses thousands of volunteers to create listings and create an organized Internet Directory owned by Netscape.
Scirus - http://scirus.com Comprehensive science-specific search engine launched by Elsevier Science.
Science.gov - http://www.science.gov Search engine created by U.S. government agencies on scientific research.
Scout Report - http://scout.wisc.edu/ Provides way to keep up-to-date with new search tools and resources.
Search Adobe PDF Online - http://searchpdf.adobe.com/ Search through more summaries of Adobe® Portable Document Format (PDF) files on the Web.
Wide Web Virtual Library - http://vlib.org/Overview.html Annotated lists of
sites compiled by experts in individual subject areas.
Naina Pandita, NIC, New Delhi