Google Crawler is Google’s robot or spider which is meant specifically to discover and scan websites automatically. While doing this it scans all the links on a website and all these links would be added to Google’s index. Google users a large number of computers to fetch billions of pages on the web and this process is done using a certain algorithmic process.
Google Crawler is commonly known as Googlebot and crawls the page based on a particular time interval. New sites, new pages, updated page content, dead link etc are all crawled and updated in the Google index. In normal case for a healthy website, Googlebot is supposed to crawl your site every second and this helps to get your new pages and contents indexed at a faster rate.
You can also block Googlebot from crawling your site or particular section of your site by making neccessary changes in the robots.txt file or in your meta tags. You can also use Google webmaster tool to find out if there is any error while Googlebot tries to crawl any section of your website. Googlebot is one of the most common Google crawler that is being known of, but there are many other Googlebot specifically for other other purposes such as crawling images, videos, news, mobile websites, Adsense etc. Here I have listed 8 common Google crawlers that is being used now a days along with the Crawler and user-agents name.
1) Googlebot – Googlebot
2) Googlebot Images – Googlebot-Image
3) Googlebot Video – Googlebot-Video
4) Googlebot News – Googlebot-News
5) Googlebot Mobile – Googlebot-Mobile
6) Google Mobile AdSense – Mediapartners-Google
7) Google AdSense – Mediapartners-Google
8) Google AdsBot – AdsBot-Google