What Googlebot does
Googlebot is the name for Google Search’s two crawlers, Googlebot Smartphone and Googlebot Desktop. Both follow the same Googlebot token in robots.txt, and most sites are crawled mainly by the smartphone crawler.
Google says blocking Googlebot affects Google Search, including Discover and all Search features. Google’s AI features guide says a page must be indexed and eligible to show a snippet to appear as a supporting link in AI Overviews or AI Mode.
Blocking crawling does not keep a URL out of results on its own. Google says to use noindex if you do not want a page indexed. Google ignores Crawl-delay.
| Operator | |
|---|---|
| robots.txt token | Googlebot |
| User agent | Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) |
| Used for | Google Search, AI Overviews and AI Mode |
| If you block it | Google cannot crawl the page for Search, including AI Overviews and AI Mode. |
| Check a visit is real | The Googlebot user agent is often spoofed. Run a reverse DNS lookup on the IP address and check it ends in googlebot.com or google.com, then a forward lookup to confirm the same IP, or match it against Google’s published IP ranges. |
Source: Google: Googlebot