Google Crawler is Google’s robot or spider which is meant specifically to discover and scan websites automatically. While doing this it scans all the links on a website and all these links would be added to Google’s index. Google users a large number of computers to fetch billions of pages on the web and this process is done using a certain algorithmic process.

Google Crawler is commonly known as Googlebot and crawls the page based on a particular time interval. New sites, new pages, updated page content, dead link etc are all crawled and updated in the Google index. In normal case for a healthy website,  Googlebot is supposed to crawl your site every second and this helps to get your new pages and contents indexed at a faster rate.
[Read more…]