What are Search Engine Spiders?

Spiders (robots, bots, crawlers) are programs employed by a search engine to explore the Internet looking for new pages and checking the content of pages they have previously visited. All the information that a search engine knows about the Internet comes from these spiders.

The best way to attract crawlers

Submitting to search engines can be an arduous task. Don't get me wrong - submit your pages to the various search engines, but don't rely on this as a sure-fire method of getting your site indexed.

The problem lies in the fact that you can't resubmit every time a page changes. Besides, search engines usually have daily submission limits (may be 5 or 10 a day). If you have a site with 20,000 pages on it, it's going to take over 5 years to submit your site. This could prove quite tedious.

The trick is to let the spiders do the work for you. Since they spend their lives looking for new pages of their own accord, why not help them out? This is where the sitemap comes in. Spiders follow all the links they can!

They follow links from pages they already know to new and undiscovered pages. All pages on the internet are linked to each other somehow, even if the link route between them may be vast and convoluted (actually, this isn't strictly true. Some pages are "Orphaned" but this is beyond the scope of this site).

...continue