Google Secrets (4) - When Google Comes Visiting
May 11th, 2008To be listed in Google’s database (or index), Google visits your site using automated
programs called robots or spiders. Such programs “read” each and every page of
your website, starting typically with your home page and then following each link to
all other web pages on your site. When a search engine robot or spider visits your
site, it is said to crawl or spider your site.
Google in the past has done two types of crawl - the deep crawl (or main crawl) and
the fresh crawl. The deep crawl is traditionally done near the end of each month and
each page is crawled. The fresh crawl is traditionally done several times a week
(daily for some sites), but only certain pages are crawled. The more “popular” your
site, the more often it typically is crawled by Google. Highly ranked sites and sites
that update content frequently (like news sites) can get crawled daily.
You can determine when your site was last crawled by looking at the date displayed
on the last line of your website’s listing on a Google search results page.
Important: Google will not add a web page to its index unless there is at least one
other web page in its index that links to one of your web pages. So don’t fret over
submitting your site to Google directly. Rather, you need to get another website to
link to your website first.