Index My Site

Bingbot: Bing's crawler, user agents and robots.txt rules

Bingbot is Microsoft’s standard crawler. It fetches pages for the Bing index, which Bing search and Copilot draw on, so a page Bingbot cannot crawl cannot be indexed in Bing.

What Bingbot does

Microsoft says Bingbot is its standard crawler and handles most of Bing’s crawling each day. It renders pages with a recent version of Microsoft Edge, which is why its user agents include the Edge version.

Bing’s robots.txt guide says a page that is not crawled will not be indexed. Bing also supports Crawl-delay, and Bing Webmaster Tools has a crawl control setting.

When Bingbot finds a group in robots.txt that names it, it ignores the generic group, so rules under User-agent: * have to be repeated in a Bingbot group.

OperatorMicrosoft
robots.txt tokenBingbot
User agentMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/W.X.Y.Z Safari/537.36Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
Used forBing and Copilot
If you block itBing cannot crawl the page for its index.
Check a visit is realRun a reverse DNS lookup on the IP address and check the name ends in search.msn.com, then a forward lookup to confirm it resolves back to the same address. Bing also publishes its IP ranges at bing.com/toolbox/bingbot.json and a Verify Bingbot tool.

Source: Bing: which crawlers Bing uses

Check whether your robots.txt allows Bingbot

We read the site's robots.txt file and nothing else. Crawl rules say who may visit. They do not show whether a page is indexed or cited.

Results

View robots.txt

How to block or allow Bingbot in robots.txt

Use the Bingbot token. Bing says it caches robots.txt for at least a few hours, so changes can take a few hours to apply.

To block Bingbot from the whole site:

User-agent: Bingbot
Disallow: /

To allow it everywhere, even when another group blocks all crawlers:

User-agent: Bingbot
Allow: /

A crawler named in a group follows only that group, so repeat any shared Disallow lines in its group. See how robots.txt blocks work for the details.

Allowed Bingbot? Get the page indexed

Bing’s index reaches further than Bing search. Microsoft describes Copilot answers that rely on it, and OpenAI’s ChatGPT Enterprise help page says ChatGPT may send searches to Bing. New pages can wait days or weeks for Bingbot, and submitting them brings it sooner.

Submit your pages to send them to Google, and add Bing to send them to Bing as well. It works for any public URL, including reviews and posts on other sites that mention you. See what each AI answer needs before it can cite a page.

Frequently asked questions

Run a reverse DNS lookup on the IP address. A real Bingbot resolves to a name ending in search.msn.com, and a forward lookup of that name returns the same IP. Bing also publishes its IP ranges and a Verify Bingbot tool.

Source: Bing: verifying Bingbot

A page Bingbot cannot crawl cannot be indexed in Bing, and Copilot draws on Bing’s index. To keep a page out of Bing and Copilot while letting Bing crawl it, Bing recommends noindex.

Yes. Bing supports Crawl-delay in robots.txt, and Bing Webmaster Tools has a crawl control setting for the same job.

When robots.txt has a group that names Bingbot, Bingbot follows only that group. Repeat any shared rules inside the Bingbot group.