Google Crawler is Google’s robot or spider which is meant specifically to discover and scan websites automatically. While doing this it scans all the links on a website and all these links would be added to Google’s index. Google users a large number of computers to fetch billions of pages on the web and this process is done using a certain algorithmic process.
Google Crawler is commonly known as Googlebot and crawls the page based on a particular time interval. New sites, new pages, updated page content, dead link etc are all crawled and updated in the Google index. In normal case for a healthy website, Googlebot is supposed to crawl your site every second and this helps to get your new pages and contents indexed at a faster rate.
You can also block Googlebot from crawling your site or particular section of your site by making neccessary changes in the robots.txt file or in your meta tags. You can also use Google webmaster tool to find out if there is any error while Googlebot tries to crawl any section of your website. Googlebot is one of the most common Google crawler that is being known of, but there are many other Googlebot specifically for other other purposes such as crawling images, videos, news, mobile websites, Adsense etc. Here I have listed 8 common Google crawlers that is being used now a days along with the Crawler and user-agents name.
1) Googlebot – Googlebot
2) Googlebot Images – Googlebot-Image
3) Googlebot Video – Googlebot-Video
4) Googlebot News – Googlebot-News
5) Googlebot Mobile – Googlebot-Mobile
6) Google Mobile AdSense – Mediapartners-Google
7) Google AdSense – Mediapartners-Google
8) Google AdsBot – AdsBot-Google
Hey George..its you old friend Neeraj. Thanks for this list and am amazed to know about these different Googlebots.. nice post..
Hey Neeraj..hw r u..? Thanks for the comments!!
Hi, Geroge.
this is vimal and i have got a problem with google search.
I am enable to see the changes in google search though the change has been made in the original site.
Actually there was an article published in one of the online journals which was made removed form the journal website. Even though this was removed from the same it still appears on google search.
I very badly want it to be removed from the google search as it created several problems.
can you heple me out how to resolve this situattion of mine?
Vimal
Hi Vimal, You can take the help of Google url removal tool offered by Google Webmasters tool https://www.google.com/webmasters/tools/removals