Index My Site

Googlebot: the crawler behind Google Search and AI Overviews

Googlebot is the crawler Google Search uses to fetch pages for its index. AI Overviews and AI Mode cite pages from that index, so Googlebot’s access decides whether Google’s AI answers can link to you.

What Googlebot does

Googlebot is the name for Google Search’s two crawlers, Googlebot Smartphone and Googlebot Desktop. Both follow the same Googlebot token in robots.txt, and most sites are crawled mainly by the smartphone crawler.

Google says blocking Googlebot affects Google Search, including Discover and all Search features. Google’s AI features guide says a page must be indexed and eligible to show a snippet to appear as a supporting link in AI Overviews or AI Mode.

Blocking crawling does not keep a URL out of results on its own. Google says to use noindex if you do not want a page indexed. Google ignores Crawl-delay.

OperatorGoogle
robots.txt tokenGooglebot
User agentMozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Used forGoogle Search, AI Overviews and AI Mode
If you block itGoogle cannot crawl the page for Search, including AI Overviews and AI Mode.
Check a visit is realThe Googlebot user agent is often spoofed. Run a reverse DNS lookup on the IP address and check it ends in googlebot.com or google.com, then a forward lookup to confirm the same IP, or match it against Google’s published IP ranges.

Source: Google: Googlebot

Check whether your robots.txt allows Googlebot

We read the site's robots.txt file and nothing else. Crawl rules say who may visit. They do not show whether a page is indexed or cited.

Results

View robots.txt

How to block or allow Googlebot in robots.txt

Use the Googlebot token. Applebot also follows your Googlebot rules when robots.txt has no Applebot group. Google caches robots.txt for up to 24 hours.

To block Googlebot from the whole site:

User-agent: Googlebot
Disallow: /

To allow it everywhere, even when another group blocks all crawlers:

User-agent: Googlebot
Allow: /

A crawler named in a group follows only that group, so repeat any shared Disallow lines in its group. See how robots.txt blocks work for the details.

Tokens that depend on your Googlebot rules

CrawlerUsed forIf you block it
Google-ExtendedGemini training and groundingGemini apps do not train on or ground answers with the page. Google Search, including AI Overviews, is unaffected.
ApplebotSiri, Spotlight and SafariApple cannot crawl the page for search across its devices.

Allowed Googlebot? Get the page indexed

Allowing Googlebot lets Google crawl, and Google still decides whether to index the page. New pages can wait days or weeks. Submitting them brings Googlebot sooner, and the Google index checker shows when they are in.

Submit your pages to send them to Google, and add Bing to send them to Bing as well. It works for any public URL, including reviews and posts on other sites that mention you. See what each AI answer needs before it can cite a page.

Frequently asked questions

Yes. AI Overviews and AI Mode are part of Google Search and cite pages from Google’s index. Google says blocking Googlebot affects all Search features.

Source: Google: Googlebot

Run a reverse DNS lookup on the IP address, check the name ends in googlebot.com or google.com, then run a forward lookup to confirm it returns the same IP. Or match the IP against Google’s published ranges.

Not with robots.txt. Google says both follow the same Googlebot token.

Not on its own. Google says a blocked URL can still appear in results. Use noindex, and let Googlebot crawl the page so it can see the noindex.