The Internet
When you go to Google’s website or your browser’s Google toolbar and type a few words in the search box, do you think you’re searching the entire Internet?
NO, YOU’RE NOT!
You’re only searching Google’s database of websites. Like many search engines, Google uses a program called a spider, bot, or crawler to “crawl” the Internet looking for web pages to include in its database. Sure, it’s really big — it contains billions of web pages!
But it’s not the entire Internet, by any means.
The same is true for other search engines. Why do you think they can find over 100,000 pages that match your search terms in 0.67 seconds? Because their enormous database of web pages is indexed on every keyword.
So the search engine looks your keywords up in the database index, compares the page ID numbers associated with those terms, and lists all of the pages that contain your search terms.
That’s the power of indexing! See my page Understanding Databases for more information about databases and indexes.
But each search engine has a different database, depending on the criteria they use to include a page in their database. There may be some overlap, but many search engines have web pages that no other search engine has. And some allow you to create a more precise search strategy than others.
So if you want to do a comprehensive search, you need to use several search engines. Yes, Google probably has the biggest market share, but there ARE other search engines!
Or you can use a search engine that automatically searches several search engine databases. These sites are often called “portals.”
Our favorite portal is Clusty. Clusty searches several search engine databases, combines the results, groups them by topic in a pane on the left side of the results page, and lists the ranked results in the main part of the page.
Do you ever look at the 20th page of your search results? How about the 45th page? Clusty brings those remote hits to the first page in the topical list on the left side of the results page.
Try it!