--- title: "Blocked by robots.txt: what it means and how to fix it" description: "Your robots.txt stops Googlebot crawling the page. See when \"Blocked by robots.txt\" is fine, how to find the rule, and how to unblock pages that should rank." canonical: "https://indexmysite.app/blocked-by-robots-txt" language: "en" --- # Blocked by robots.txt: find the rule and unblock pages that should rank Your robots.txt file tells Googlebot not to crawl the page. Often that is on purpose. Here is how to tell intended blocks from mistakes, find the rule that matches, and let Google crawl the pages you want in search. [Check a page for blocks](#check)[Submit fixed pages](https://app.indexmysite.app/signup) ## What the status means In the Page indexing report, "Blocked by robots.txt" means a rule in your robots.txt file stopped Googlebot from crawling the URL, so Google did not fetch the page. Google did not fetch the page Google knows the URL but obeyed your robots.txt and did not crawl it. It has not seen the content. It controls crawling, not indexing If other pages link to the URL, Google can still index it without its content. That shows as "Indexed, though blocked by robots.txt". It is often intended Google notes that URLs blocked by robots.txt are probably blocked on purpose. It only matters for pages that should rank. Definition from Google's [Page indexing report help page](https://support.google.com/webmasters/answer/7440203). ## Why robots.txt blocks the page Some blocks are deliberate. The rest are usually a leftover file, a rule broader than intended, or a group you forgot about. - You meant to block it Admin areas, carts, internal search results, and endless filter or sort URLs are often blocked on purpose to save crawling. Leave these alone. - A "Disallow: /" was left over from staging A staging site often blocks every crawler with "User-agent: \*" and "Disallow: /". If that file ships to production, Google stops crawling the whole site. - A broad rule catches real pages Rules match the start of the path. "Disallow: /p" blocks /pricing and /products as well as /p. Wildcards like "Disallow: /\*?" can block every URL with a query string. - A Googlebot group overrides your general rules Google follows only the most specific group that matches it. If there is a "User-agent: Googlebot" group, Googlebot ignores everything under "User-agent: \*". - CSS or JavaScript files are blocked Rules on folders like /assets/ or /static/ can stop Google loading the files your pages need, so it cannot render them properly. - The rule is on a different host Each host and protocol has its own robots.txt. A page on docs.example.com follows docs.example.com/robots.txt, not the file on example.com. ## Check if robots.txt blocks the page Paste the URL from Search Console. The free checker reads the site's robots.txt and reports whether its rules block Googlebot from the page. It also looks for noindex rules and other indexing settings. Use the free indexability checker at /tools/indexability. Enter a public page address and it reports whether the site lets Google visit the page, whether the page carries a noindex setting or points to another page as the original, and whether the content is thin, repeated, or unclear. It does not tell you whether Google has indexed the page. ## How to unblock a page in robots.txt First decide whether Google should crawl the page. If it should, change the rule that blocks it, confirm the change in Search Console, and submit the URL last. 1. 1 Decide if Google should crawl the page If the URL is an admin page, cart, internal search result, or filter combination, the block is probably correct. Only keep going for pages that should rank. 2. 2 Find the rule that blocks it Open robots.txt on the same host as the page, for example https://example.com/robots.txt. Find the group that applies to Googlebot and the Disallow rule that matches the path. URL Inspection in Search Console also tells you if the URL is blocked. 3. 3 Change the rule as narrowly as you can Delete the rule, tighten it, or add a more specific Allow rule for the pages you want crawled. Google applies the most specific matching rule, and when rules conflict it uses the least restrictive one. 4. 4 Unblock the files pages need Make sure the CSS, JavaScript, and images your pages depend on are not disallowed, so Google can render the page the way visitors see it. 5. 5 Check the change in Search Console Open the robots.txt report to see the version Google last fetched and any parse errors. After an important change, use Request a recrawl there. Then run a live test in URL Inspection on the page. 6. 6 Then submit the URL Once Google can crawl the page, request indexing in URL Inspection, or submit the URL with Index My Site so Googlebot fetches it. ## When submitting helps, and when it will not Submitting asks Googlebot to crawl the page. While robots.txt blocks the URL, Googlebot will not fetch it, so the rule has to change first. It helps when - You removed or narrowed the rule and URL Inspection shows the page can be crawled - You replaced a staging robots.txt that blocked the whole site - You unblocked a folder and want its important pages crawled first - You want the page out of Google, so you unblocked it and added noindex It will not help when - The rule that blocks Googlebot is still in robots.txt - You edited the wrong file, such as the one on another subdomain - The page also has a noindex tag you have not removed - The page is meant to stay uncrawled, like a cart or admin page When you submit a URL, we check it first and skip redirects, noindex pages, robots.txt blocks, error pages, and pages that point to another page as the original, without charging. Then we help Googlebot discover and crawl it and search Google for the exact URL after 24 hours, again around days 3 and 7 if it is not found yet, with a full report around day 5and final checks around day 14. A URL confirmed as not indexed gets its standard credit back. Google still decides what it indexes. See [Submitting URLs](https://indexmysite.app/docs/submitting-urls) for the details. ## robots.txt vs noindex They do different jobs. robots.txt controls whether Google crawls a URL. noindex controls whether a page appears in search results. robots.txt compared with noindex | Question | Excluded by 'noindex' tag | Blocked by robots.txt | | --- | --- | --- | | What it controls | Indexing: whether Google may show the page in search results | Crawling: whether Googlebot may fetch the URL | | Where it is set | A robots meta tag in the page head, or an X-Robots-Tag HTTP response header | One robots.txt file at the root of the host, such as example.com/robots.txt | | Can Google see the page? | Yes. Google has to crawl the page to find the noindex | No. Google does not fetch it, so it cannot read the content or any noindex on it | | Can the URL still appear in results? | No. Once Google sees the noindex, it drops the page from search results | Yes. If other pages link to it, Google can index the URL without its content, usually with little or no description | | Use it when | You want a page kept out of Google search results | You want to stop Google spending crawls on URLs such as endless filter or parameter pages | ## Other Page indexing statuses Each reason in the report needs a different fix. Find the one Search Console shows you. [ ### Discovered - currently not indexed Google found the URL but has not crawled it yet. ](https://indexmysite.app/discovered-currently-not-indexed)[ ### Crawled - currently not indexed Google crawled the page and chose not to index it for now. ](https://indexmysite.app/crawled-currently-not-indexed)[ ### Duplicate without user-selected canonical The page does not name a main version, so Google picked another URL as the original. ](https://indexmysite.app/duplicate-without-user-selected-canonical)[ ### Duplicate, Google chose different canonical than user The page names itself as the main version, but Google picked another URL instead. ](https://indexmysite.app/duplicate-google-chose-different-canonical-than-user)[ ### Alternate page with proper canonical tag The page points to another page as the main version, and Google agreed. ](https://indexmysite.app/alternate-page-with-proper-canonical-tag)[ ### Soft 404 The page loads, but Google thinks it looks like an error or empty page. ](https://indexmysite.app/soft-404)[ ### Excluded by 'noindex' tag The page tells Google not to index it. ](https://indexmysite.app/excluded-by-noindex-tag)[ ### Page with redirect The URL redirects to another page, so Google does not index this URL. ](https://indexmysite.app/page-with-redirect) ## Tools for getting pages indexed [ ### Every Page indexing status What each reason in the report means, whether it needs fixing, and whether submitting helps. ](https://indexmysite.app/page-indexing-report)[ ### Free indexability checker The full checker page, with notes on what each result means and what to do next. ](https://indexmysite.app/tools/indexability)[ ### Indexing API Submit fixed pages from your own code with one API key and track each one until it is indexed. ](https://indexmysite.app/indexing-api)[ ### Automatic indexing Submit pages every time you publish or fix them, from your sitemap, your deploy, or your agent. ](https://indexmysite.app/automatic-indexing) ## Frequently asked questions ### What does "Blocked by robots.txt" mean? Your site's robots.txt file has a rule that stops Googlebot from crawling the URL, so Google did not fetch the page. Google notes that URLs blocked by robots.txt are probably blocked on purpose, so check whether the page should be crawled before you change anything. Source: [Google Search Console Help: Page indexing report](https://support.google.com/webmasters/answer/7440203) ### Can a page blocked by robots.txt still show up in Google? Yes. robots.txt stops crawling, not indexing. If other pages link to the URL, Google can index it without visiting it, using information such as the link text. Search Console then lists it as "Indexed, though blocked by robots.txt", and the result usually has little or no description. Source: [Google Search Console Help: Page indexing report](https://support.google.com/webmasters/answer/7440203) ### Should I use robots.txt to keep a page out of Google? No. Google says robots.txt is not a way to keep a page out of Google. Use a noindex meta tag or header, or put the page behind a password. If you add noindex, also remove the robots.txt block, because Google has to crawl the page to see the noindex. Source: [Google Search Central: Introduction to robots.txt](https://developers.google.com/search/docs/crawling-indexing/robots/intro) ### How do I find which rule blocks my page? Open robots.txt on the same host and protocol as the page. Googlebot follows only the most specific user-agent group that matches it, so check the Googlebot group if there is one, otherwise the "User-agent: \*" group. Within that group, the longest matching rule wins, and rules are case-sensitive. Source: [Google Search Central: How Google interprets the robots.txt specification](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt) ### I fixed robots.txt. How long until Google notices? Google generally caches robots.txt for up to 24 hours. After an important change, use Request a recrawl in the robots.txt report in Search Console. Google says that does not guarantee the unblocked URLs are crawled right away, so submit the pages that matter too. Source: [Google Search Console Help: robots.txt report](https://support.google.com/webmasters/answer/6062598) ### What does Index My Site do with a submitted URL? We check the page for redirects, noindex tags, robots.txt blocks, and error pages first. A URL that robots.txt still blocks for Googlebot is skipped without charging, so fix the rule before you submit. Then we help Googlebot discover and crawl the page. We check every URL once after 24 hours. Your full report follows around day 5, with final checks around day 14. If Google has not indexed the URL at the final check, the standard credit comes back. ### What does it cost? Each sent URL costs 1 credit, which is $0.20, or $0.10 on purchases of $1,000 or more. URLs skipped before sending, including pages robots.txt blocks for Googlebot, are not charged. ## Unblocked the page? Get Google to crawl it Submit fixed pages from the web app, the API, the CLI, or your AI agent. Credits start at $25. [Submit fixed pages](https://app.indexmysite.app/signup) [See pricing](https://indexmysite.app/pricing)