Index My Site opens each page before sending it for indexing. This check can
find redirects, crawl blocks, noindex rules, error pages and canonicals that
point elsewhere. A page is not charged while it is being checked.
Our crawler identifies itself with this user agent:
IndexMySiteChecker/1.0 (+https://indexmysite.app/docs/allow-crawler)If your firewall blocks the check, allow this user agent, then choose Check again on the job. We do not check blocked pages again on our own, and you do not need to submit the URLs again. You can also choose Submit without checks on the existing job. That sends the affected URLs without confirming that they are eligible for indexing. Each URL uses one credit when it is sent.
Allow it in Cloudflare
In the Cloudflare dashboard for your site:
- Open Security, then WAF, and create a custom rule.
- Limit the rule to your hostname and, if practical, the paths you submit.
- Match requests whose User Agent contains
IndexMySiteChecker. - Choose Skip and select the security features that blocked the request.
- Save the rule, then choose Check again on the job in Index My Site.
An example expression for one hostname is:
(http.host eq "www.example.com" and http.user_agent contains "IndexMySiteChecker")Cloudflare may label or arrange these settings differently by plan. Check the security events for the blocked request to see which feature to skip. Make the rule as narrow as possible. A user agent can be copied by someone else, so do not use it to bypass sign-in or protect private information.
Our page checks run on managed cloud infrastructure and do not have one fixed IP address to allow.
Allow it in robots.txt
If the job says your robots.txt blocks our checker, add:
User-agent: IndexMySiteChecker
Allow: /Preserve any rules for Googlebot that match your intended indexing policy. Allowing our checker does not override a Googlebot block, and it does not guarantee that Google will index a page.
Check that it worked
Choose Check again on the job. The URLs should move past Checking page without showing Our crawler was blocked. If it is still blocked, review the latest firewall event and confirm that the rule matches the hostname and path.