Blocked by robots.txt: find the rule and unblock pages that should rank
Your robots.txt file tells Googlebot not to crawl the page. Often that is on purpose. Here is how to tell intended blocks from mistakes, find the rule that matches, and let Google crawl the pages you want in search.
What the status means
In the Page indexing report, "Blocked by robots.txt" means a rule in your robots.txt file stopped Googlebot from crawling the URL, so Google did not fetch the page.
Google did not fetch the page
Google knows the URL but obeyed your robots.txt and did not crawl it. It has not seen the content.
It controls crawling, not indexing
If other pages link to the URL, Google can still index it without its content. That shows as "Indexed, though blocked by robots.txt".
It is often intended
Google notes that URLs blocked by robots.txt are probably blocked on purpose. It only matters for pages that should rank.
Definition from Google's Page indexing report help page.
Why robots.txt blocks the page
Some blocks are deliberate. The rest are usually a leftover file, a rule broader than intended, or a group you forgot about.
You meant to block it
Admin areas, carts, internal search results, and endless filter or sort URLs are often blocked on purpose to save crawling. Leave these alone.
A "Disallow: /" was left over from staging
A staging site often blocks every crawler with "User-agent: *" and "Disallow: /". If that file ships to production, Google stops crawling the whole site.
A broad rule catches real pages
Rules match the start of the path. "Disallow: /p" blocks /pricing and /products as well as /p. Wildcards like "Disallow: /*?" can block every URL with a query string.
A Googlebot group overrides your general rules
Google follows only the most specific group that matches it. If there is a "User-agent: Googlebot" group, Googlebot ignores everything under "User-agent: *".
CSS or JavaScript files are blocked
Rules on folders like /assets/ or /static/ can stop Google loading the files your pages need, so it cannot render them properly.
The rule is on a different host
Each host and protocol has its own robots.txt. A page on docs.example.com follows docs.example.com/robots.txt, not the file on example.com.
Check if robots.txt blocks the page
Paste the URL from Search Console. The free checker reads the site's robots.txt and reports whether its rules block Googlebot from the page. It also looks for noindex rules and other indexing settings.
Progress
Checks that completed before it stopped are shown below.
Access and indexing settings
Content review
Already in Google?
Not checked
This check cannot confirm whether your page is already in Google, and it does not predict the outcome. It shows what could stand in the way.
· this address redirects to , which was not checked. Run a separate check on it.
Only part of this long page was reviewed.
Good to know
Completed before the check stopped
Use the free indexability checker at /tools/indexability. Enter a public page address and it reports whether the site lets Google visit the page, whether the page carries a noindex setting or points to another page as the original, and whether the content is thin, repeated, or unclear. It does not tell you whether Google has indexed the page.
How to unblock a page in robots.txt
First decide whether Google should crawl the page. If it should, change the rule that blocks it, confirm the change in Search Console, and submit the URL last.
- 1
Decide if Google should crawl the page
If the URL is an admin page, cart, internal search result, or filter combination, the block is probably correct. Only keep going for pages that should rank.
- 2
Find the rule that blocks it
Open robots.txt on the same host as the page, for example https://example.com/robots.txt. Find the group that applies to Googlebot and the Disallow rule that matches the path. URL Inspection in Search Console also tells you if the URL is blocked.
- 3
Change the rule as narrowly as you can
Delete the rule, tighten it, or add a more specific Allow rule for the pages you want crawled. Google applies the most specific matching rule, and when rules conflict it uses the least restrictive one.
- 4
Unblock the files pages need
Make sure the CSS, JavaScript, and images your pages depend on are not disallowed, so Google can render the page the way visitors see it.
- 5
Check the change in Search Console
Open the robots.txt report to see the version Google last fetched and any parse errors. After an important change, use Request a recrawl there. Then run a live test in URL Inspection on the page.
- 6
Then submit the URL
Once Google can crawl the page, request indexing in URL Inspection, or submit the URL with Index My Site so Googlebot fetches it.
When submitting helps, and when it will not
Submitting asks Googlebot to crawl the page. While robots.txt blocks the URL, Googlebot will not fetch it, so the rule has to change first.
It helps when
- You removed or narrowed the rule and URL Inspection shows the page can be crawled
- You replaced a staging robots.txt that blocked the whole site
- You unblocked a folder and want its important pages crawled first
- You want the page out of Google, so you unblocked it and added noindex
It will not help when
- The rule that blocks Googlebot is still in robots.txt
- You edited the wrong file, such as the one on another subdomain
- The page also has a noindex tag you have not removed
- The page is meant to stay uncrawled, like a cart or admin page
When you submit a URL, we check it first and skip redirects, noindex pages, robots.txt blocks, error pages, and pages that point to another page as the original, without charging. Then we help Googlebot discover and crawl it and search Google for the exact URL after 24 hours, again around days 3 and 7 if it is not found yet, with a full report around day 5and final checks around day 14. A URL confirmed as not indexed gets its standard credit back. Google still decides what it indexes. See Submitting URLs for the details.
robots.txt vs noindex
They do different jobs. robots.txt controls whether Google crawls a URL. noindex controls whether a page appears in search results.
| Question | Excluded by 'noindex' tag | Blocked by robots.txt |
|---|---|---|
| What it controls | Indexing: whether Google may show the page in search results | Crawling: whether Googlebot may fetch the URL |
| Where it is set | A robots meta tag in the page head, or an X-Robots-Tag HTTP response header | One robots.txt file at the root of the host, such as example.com/robots.txt |
| Can Google see the page? | Yes. Google has to crawl the page to find the noindex | No. Google does not fetch it, so it cannot read the content or any noindex on it |
| Can the URL still appear in results? | No. Once Google sees the noindex, it drops the page from search results | Yes. If other pages link to it, Google can index the URL without its content, usually with little or no description |
| Use it when | You want a page kept out of Google search results | You want to stop Google spending crawls on URLs such as endless filter or parameter pages |
- What it controls
- Excluded by 'noindex' tagIndexing: whether Google may show the page in search results
- Blocked by robots.txtCrawling: whether Googlebot may fetch the URL
- Where it is set
- Excluded by 'noindex' tagA robots meta tag in the page head, or an X-Robots-Tag HTTP response header
- Blocked by robots.txtOne robots.txt file at the root of the host, such as example.com/robots.txt
- Can Google see the page?
- Excluded by 'noindex' tagYes. Google has to crawl the page to find the noindex
- Blocked by robots.txtNo. Google does not fetch it, so it cannot read the content or any noindex on it
- Can the URL still appear in results?
- Excluded by 'noindex' tagNo. Once Google sees the noindex, it drops the page from search results
- Blocked by robots.txtYes. If other pages link to it, Google can index the URL without its content, usually with little or no description
- Use it when
- Excluded by 'noindex' tagYou want a page kept out of Google search results
- Blocked by robots.txtYou want to stop Google spending crawls on URLs such as endless filter or parameter pages
Other Page indexing statuses
Each reason in the report needs a different fix. Find the one Search Console shows you.
Discovered - currently not indexed
Google found the URL but has not crawled it yet.
Crawled - currently not indexed
Google crawled the page and chose not to index it for now.
Duplicate without user-selected canonical
The page does not name a main version, so Google picked another URL as the original.
Duplicate, Google chose different canonical than user
The page names itself as the main version, but Google picked another URL instead.
Alternate page with proper canonical tag
The page points to another page as the main version, and Google agreed.
Soft 404
The page loads, but Google thinks it looks like an error or empty page.
Excluded by 'noindex' tag
The page tells Google not to index it.
Page with redirect
The URL redirects to another page, so Google does not index this URL.
Tools for getting pages indexed
Every Page indexing status
What each reason in the report means, whether it needs fixing, and whether submitting helps.
Free indexability checker
The full checker page, with notes on what each result means and what to do next.
Indexing API
Submit fixed pages from your own code with one API key and track each one until it is indexed.
Automatic indexing
Submit pages every time you publish or fix them, from your sitemap, your deploy, or your agent.
Frequently asked questions
Your site's robots.txt file has a rule that stops Googlebot from crawling the URL, so Google did not fetch the page. Google notes that URLs blocked by robots.txt are probably blocked on purpose, so check whether the page should be crawled before you change anything.
Yes. robots.txt stops crawling, not indexing. If other pages link to the URL, Google can index it without visiting it, using information such as the link text. Search Console then lists it as "Indexed, though blocked by robots.txt", and the result usually has little or no description.
No. Google says robots.txt is not a way to keep a page out of Google. Use a noindex meta tag or header, or put the page behind a password. If you add noindex, also remove the robots.txt block, because Google has to crawl the page to see the noindex.
Open robots.txt on the same host and protocol as the page. Googlebot follows only the most specific user-agent group that matches it, so check the Googlebot group if there is one, otherwise the "User-agent: *" group. Within that group, the longest matching rule wins, and rules are case-sensitive.
Source: Google Search Central: How Google interprets the robots.txt specification
Google generally caches robots.txt for up to 24 hours. After an important change, use Request a recrawl in the robots.txt report in Search Console. Google says that does not guarantee the unblocked URLs are crawled right away, so submit the pages that matter too.
We check the page for redirects, noindex tags, robots.txt blocks, and error pages first. A URL that robots.txt still blocks for Googlebot is skipped without charging, so fix the rule before you submit. Then we help Googlebot discover and crawl the page. We check every URL once after 24 hours. Your full report follows around day 5, with final checks around day 14. If Google has not indexed the URL at the final check, the standard credit comes back.
Each sent URL costs 1 credit, which is $0.20, or $0.10 on purchases of $1,000 or more. URLs skipped before sending, including pages robots.txt blocks for Googlebot, are not charged.
What does "Blocked by robots.txt" mean?
Your site's robots.txt file has a rule that stops Googlebot from crawling the URL, so Google did not fetch the page. Google notes that URLs blocked by robots.txt are probably blocked on purpose, so check whether the page should be crawled before you change anything.
Can a page blocked by robots.txt still show up in Google?
Yes. robots.txt stops crawling, not indexing. If other pages link to the URL, Google can index it without visiting it, using information such as the link text. Search Console then lists it as "Indexed, though blocked by robots.txt", and the result usually has little or no description.
Should I use robots.txt to keep a page out of Google?
No. Google says robots.txt is not a way to keep a page out of Google. Use a noindex meta tag or header, or put the page behind a password. If you add noindex, also remove the robots.txt block, because Google has to crawl the page to see the noindex.
How do I find which rule blocks my page?
Open robots.txt on the same host and protocol as the page. Googlebot follows only the most specific user-agent group that matches it, so check the Googlebot group if there is one, otherwise the "User-agent: *" group. Within that group, the longest matching rule wins, and rules are case-sensitive.
Source: Google Search Central: How Google interprets the robots.txt specification
I fixed robots.txt. How long until Google notices?
Google generally caches robots.txt for up to 24 hours. After an important change, use Request a recrawl in the robots.txt report in Search Console. Google says that does not guarantee the unblocked URLs are crawled right away, so submit the pages that matter too.
What does Index My Site do with a submitted URL?
We check the page for redirects, noindex tags, robots.txt blocks, and error pages first. A URL that robots.txt still blocks for Googlebot is skipped without charging, so fix the rule before you submit. Then we help Googlebot discover and crawl the page. We check every URL once after 24 hours. Your full report follows around day 5, with final checks around day 14. If Google has not indexed the URL at the final check, the standard credit comes back.
What does it cost?
Each sent URL costs 1 credit, which is $0.20, or $0.10 on purchases of $1,000 or more. URLs skipped before sending, including pages robots.txt blocks for Googlebot, are not charged.
Unblocked the page? Get Google to crawl it
Submit fixed pages from the web app, the API, the CLI, or your AI agent. Credits start at $25.