Index My Site

How Bing finds, crawls and indexes your pages

Bing’s index serves Bing search, Copilot and grounding results. See how Bingbot gets a page into it, the common reasons a page gets skipped, and how to check where a URL stands.

From discovery to the Bing index

Every page goes through the same three stages before it can appear in Bing or Copilot.

  1. Step 1

    Discover

    Bing finds URLs through IndexNow, XML sitemaps, links within your site and links from other relevant sites.

  2. Step 2

    Crawl and render

    Bingbot fetches the page and renders it with a recent version of Microsoft Edge. Bing shares crawl capacity across a site based on its health and how many useful URLs it has.

  3. Step 3

    Index

    Bing decides whether to keep the page, picks one URL among duplicates, and adds it to the index behind Bing search, Copilot and grounding results.

From the Bing Webmaster Guidelines and Bing’s crawler overview.

Why Bing skips pages, and the fix for each

These are the causes Bing’s own guidelines describe, with what to change.

  • robots.txt blocks Bingbot

    Allow Bingbot for the path. Check it with the crawler checker below.

  • The page has a noindex rule

    Bing removes noindex pages from Bing, Copilot and grounding results. Remove the rule from pages you want found.

  • It duplicates another URL

    Bing picks one URL among duplicates. Use a canonical and consistent URLs so it picks the one you want.

  • The content loads in the browser

    Bing may skip content it cannot render reliably. Put the main text in the HTML the server sends.

  • The page is thin or mostly ads

    Bing says thin, ad-heavy or affiliate-only pages may lose eligibility. Make the page fully answer what people search for.

  • Bing has yet to find it

    New pages with no links and no sitemap entry wait longest. Link to them, list them in your sitemap, and send them to Bing.

  • The site wastes crawl capacity

    Large numbers of duplicate or parameter URLs delay important pages. Consolidate them and block crawl paths with no value.

  • Send pages to Bing and Google

    Index My Site sends any public URL to Bing and Google in one job, 2 credits per URL.

    Submit a URL to Bing

Check Bingbot can reach a page

Paste a page address to see whether robots.txt lets Bingbot, Googlebot and the AI search crawlers visit it, and the rule that decides each one.

We read the site's robots.txt file and nothing else. Crawl rules say who may visit. They do not show whether a page is indexed or cited.

Results

View robots.txt

How to check if Bing indexed a page

URL Inspection in Bing Webmaster Tools gives the status of any URL on a site you have verified.

  • Index card

    Shows each step from discovery to indexing, with dates, the HTTP response and any error and its fix.

  • Live URL

    Fetches the page as Bingbot now, so you can see the HTML Bing gets and spot anything blocking it.

From Bing’s URL inspection help.

Bingbot and the other Bing crawlers

Bingbot does most of Bing’s crawling. To confirm a visit is real, check that a reverse DNS lookup of its IP ends in search.msn.com and a forward lookup returns the same IP.

  • Bingbot

    The standard crawler that fetches pages for the Bing index. Most Bing crawling is Bingbot.

  • AdIdxBot

    Crawls Bing Ads and the sites they link to, for quality control.

  • BingPreview

    Makes page snapshots for Bing.

  • MicrosoftPreview

    Makes page snapshots for other Microsoft products.

From Bing’s crawler overview and how to verify Bingbot.

Frequently asked questions

The usual causes are a robots.txt rule that blocks Bingbot, a noindex rule, duplicate URLs, content that only appears after JavaScript runs, thin pages, or pages Bing has yet to find. URL Inspection in Bing Webmaster Tools shows the status of a URL on a verified site and suggests the fix.

Source: Bing Webmaster Tools: URL inspection

Bing’s standard web crawler. It handles most of Bing’s crawling and identifies itself with bingbot/2.0 and a link to bing.com/bingbot.htm in its user agent. Bing also runs AdIdxBot for ads and BingPreview and MicrosoftPreview for page snapshots.

Source: Bing: which crawlers Bing uses

Run a reverse DNS lookup on the visiting IP address and check that the name ends in search.msn.com, then a forward lookup on that name to confirm it returns the same IP. Bing also publishes a Verify Bingbot tool and a daily list of Bingbot IP addresses.

Source: Bing: verifying Bingbot

robots.txt controls crawling. To keep a URL out of Bing search, Copilot and grounding results, Bing says to use noindex, and to return a 404 when a page is gone for good.

Source: Bing Webmaster Guidelines

Yes. Bing says Bing and Copilot search experiences rely on the same crawling, indexing and ranking foundation, and its guidelines cover eligibility for Copilot answers and grounding citations.

Source: Bing Webmaster Guidelines

For a site you have verified, URL Inspection in Bing Webmaster Tools shows each step Bing has taken with a URL, from discovery to indexing. Its Live URL tab shows what Bingbot sees when it fetches the page.

Source: Bing Webmaster Tools: URL inspection

Yes. Use robots.txt to choose what Bing crawls, and the Crawl control tool in Bing Webmaster Tools to set crawl rates by the hour.

Source: Bing: which crawlers Bing uses

Get your pages into Bing and Google

Create an account, buy credits from $25, and send any public URL to both in one job.