Index My Site

PerplexityBot: how Perplexity crawls and cites pages

PerplexityBot is the crawler Perplexity uses to surface and link websites in its search results. Perplexity says it is not used to train AI models, and recommends allowing it if you want your site to appear.

What PerplexityBot does

Perplexity says PerplexityBot is designed to surface and link websites in search results on Perplexity, and is not used to crawl content for AI foundation models.

Perplexity-User is a separate fetcher that visits a page when a person asks Perplexity a question that needs it. Perplexity says it generally ignores robots.txt, because a person requested the fetch.

Perplexity says changes to its crawler settings can take up to 24 hours to apply.

OperatorPerplexity
robots.txt tokenPerplexityBot
User agentMozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)
Used forPerplexity search
If you block itPerplexity does not surface and link the page in its search results.
Check a visit is realPerplexity publishes PerplexityBot’s IP addresses at perplexity.ai/perplexitybot.json and recommends firewall rules that match both the user agent and the IP address.

Source: Perplexity: Perplexity crawlers

Check whether your robots.txt allows PerplexityBot

We read the site's robots.txt file and nothing else. Crawl rules say who may visit. They do not show whether a page is indexed or cited.

Results

View robots.txt

How to block or allow PerplexityBot in robots.txt

Use the PerplexityBot token. Perplexity’s docs include allow rules for common firewalls, and note that some bot protection modes cannot be skipped with custom rules.

To block PerplexityBot from the whole site:

User-agent: PerplexityBot
Disallow: /

To allow it everywhere, even when another group blocks all crawlers:

User-agent: PerplexityBot
Allow: /

A crawler named in a group follows only that group, so repeat any shared Disallow lines in its group. See how robots.txt blocks work for the details.

Perplexity's other crawler

CrawlerUsed forIf you block it
Perplexity-UserPerplexity, when a person asks it to visit a pagePerplexity says these visits generally ignore robots.txt.

Allowed PerplexityBot? Get the page indexed

Perplexity has no submission form. Its help center says it also works with third-party crawlers to build its index, so a page that is linked from known pages and indexed in the big search engines is easier for it to find.

Submit your pages to send them to Google, and add Bing to send them to Bing as well. It works for any public URL, including reviews and posts on other sites that mention you. See what each AI answer needs before it can cite a page.

Read next: How Perplexity finds and cites pages.

Frequently asked questions

Allow PerplexityBot in robots.txt and its published IP addresses at your firewall. Perplexity has no submission form, so link new pages from pages its crawler already knows.

No. Perplexity says PerplexityBot is not used to crawl content for AI foundation models.

Those visits are probably Perplexity-User, which fetches a page when a person asks for it. Perplexity says that fetcher generally ignores robots.txt.

Perplexity says it may take up to 24 hours for its systems to reflect a change.