---
title: "Googlebot: the crawler behind Google Search and AI Overviews"
description: "Googlebot crawls pages for Google Search, including AI Overviews and AI Mode. Its user agents, how to verify it, and what blocking it in robots.txt does."
canonical: "https://indexmysite.app/tools/ai-crawler-checker/googlebot"
language: "en"
---

# Googlebot: the crawler behind Google Search and AI Overviews

Googlebot is the crawler Google Search uses to fetch pages for its index. AI Overviews and AI Mode cite pages from that index, so Googlebot’s access decides whether Google’s AI answers can link to you.

## What Googlebot does

Googlebot is the name for Google Search’s two crawlers, Googlebot Smartphone and Googlebot Desktop. Both follow the same Googlebot token in robots.txt, and most sites are crawled mainly by the smartphone crawler.

Google says blocking Googlebot affects Google Search, including Discover and all Search features. Google’s AI features guide says a page must be indexed and eligible to show a snippet to appear as a supporting link in AI Overviews or AI Mode.

Blocking crawling does not keep a URL out of results on its own. Google says to use noindex if you do not want a page indexed. Google ignores Crawl-delay.

<table><tbody><tr><th scope="row">Operator</th><td>Google</td></tr><tr><th scope="row">robots.txt token</th><td><code>Googlebot</code></td></tr><tr><th scope="row">User agent</th><td><code class="block break-all">Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)</code><code class="block break-all">Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36</code><code class="block break-all">Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)</code></td></tr><tr><th scope="row">Used for</th><td>Google Search, AI Overviews and AI Mode</td></tr><tr><th scope="row">If you block it</th><td>Google cannot crawl the page for Search, including AI Overviews and AI Mode.</td></tr><tr><th scope="row">Check a visit is real</th><td>The Googlebot user agent is often spoofed. Run a reverse DNS lookup on the IP address and check it ends in googlebot.com or google.com, then a forward lookup to confirm the same IP, or match it against Google’s published IP ranges.</td></tr></tbody></table>

Source: [Google: Googlebot](https://developers.google.com/search/docs/crawling-indexing/googlebot)

## Check whether your robots.txt allows Googlebot

Use the [AI crawler checker](https://indexmysite.app/tools/ai-crawler-checker) to read a site's robots.txt and see whether Googlebot and the other search, AI and training crawlers may visit a page.

## How to block or allow Googlebot in robots.txt

Use the Googlebot token. Applebot also follows your Googlebot rules when robots.txt has no Applebot group. Google caches robots.txt for up to 24 hours.

To block Googlebot from the whole site:

```
User-agent: Googlebot
Disallow: /
```

To allow it everywhere, even when another group blocks all crawlers:

```
User-agent: Googlebot
Allow: /
```

A crawler named in a group follows only that group, so repeat any shared Disallow lines in its group. See [how robots.txt blocks work](https://indexmysite.app/blocked-by-robots-txt) for the details.

## Tokens that depend on your Googlebot rules

| Crawler | Used for | If you block it |
| --- | --- | --- |
| [`Google-Extended`](https://indexmysite.app/tools/ai-crawler-checker/google-extended) | Gemini training and grounding | Gemini apps do not train on or ground answers with the page. Google Search, including AI Overviews, is unaffected. |
| [`Applebot`](https://indexmysite.app/tools/ai-crawler-checker/applebot) | Siri, Spotlight and Safari | Apple cannot crawl the page for search across its devices. |

## Allowed Googlebot? Get the page indexed

Allowing Googlebot lets Google crawl, and Google still decides whether to index the page. New pages can wait days or weeks. Submitting them brings Googlebot sooner, and the Google index checker shows when they are in.

[Submit your pages](https://app.indexmysite.app/signup) to send them to Google, and add Bing to send them to Bing as well. It works for any public URL, including reviews and posts on other sites that mention you. See [what each AI answer needs](https://indexmysite.app/ai-search-indexing) before it can cite a page.

## Frequently asked questions

### Does blocking Googlebot remove me from AI Overviews?

Yes. AI Overviews and AI Mode are part of Google Search and cite pages from Google’s index. Google says blocking Googlebot affects all Search features.

Source: [Google: Googlebot](https://developers.google.com/search/docs/crawling-indexing/googlebot)

### How do I verify Googlebot?

Run a reverse DNS lookup on the IP address, check the name ends in googlebot.com or google.com, then run a forward lookup to confirm it returns the same IP. Or match the IP against Google’s published ranges.

### Can I block Googlebot Smartphone and allow Googlebot Desktop?

Not with robots.txt. Google says both follow the same Googlebot token.

### Does robots.txt keep a page out of Google?

Not on its own. Google says a blocked URL can still appear in results. Use noindex, and let Googlebot crawl the page so it can see the noindex.
