---
title: "Allow the Index My Site page checker | Index My Site"
description: "How to allow IndexMySiteChecker through Cloudflare and robots.txt before submitting URLs for indexing."
canonical: "https://indexmysite.app/docs/allow-crawler"
language: "en"
---

# Allow our page checker

How to allow IndexMySiteChecker through Cloudflare and robots.txt before submitting URLs for indexing.

Index My Site opens each page before sending it for indexing. This check can find redirects, crawl blocks, `noindex` rules, error pages and canonicals that point elsewhere. A page is not charged while it is being checked.

Our crawler identifies itself with this user agent:

```
IndexMySiteChecker/1.0 (+https://indexmysite.app/docs/allow-crawler)
```

If your firewall blocks the check, allow this user agent, then choose **Check again** on the job. We do not check blocked pages again on our own, and you do not need to submit the URLs again. You can also choose **Submit without checks** on the existing job. That sends the affected URLs without confirming that they are eligible for indexing. Each URL uses one credit when it is sent.

## Allow it in Cloudflare

In the Cloudflare dashboard for your site:

1.  Open **Security**, then **WAF**, and create a custom rule.
2.  Limit the rule to your hostname and, if practical, the paths you submit.
3.  Match requests whose **User Agent contains** `IndexMySiteChecker`.
4.  Choose **Skip** and select the security features that blocked the request.
5.  Save the rule, then choose **Check again** on the job in Index My Site.

An example expression for one hostname is:

```
(http.host eq "www.example.com" and http.user_agent contains "IndexMySiteChecker")
```

Cloudflare may label or arrange these settings differently by plan. Check the security events for the blocked request to see which feature to skip. Make the rule as narrow as possible. A user agent can be copied by someone else, so do not use it to bypass sign-in or protect private information.

Our page checks run on managed cloud infrastructure and do not have one fixed IP address to allow.

## Allow it in robots.txt

If the job says your `robots.txt` blocks our checker, add:

```
User-agent: IndexMySiteChecker
Allow: /
```

Preserve any rules for Googlebot that match your intended indexing policy. Allowing our checker does not override a Googlebot block, and it does not guarantee that Google will index a page.

## Check that it worked

Choose **Check again** on the job. The URLs should move past **Checking page** without showing **Our crawler was blocked**. If it is still blocked, review the latest firewall event and confirm that the rule matches the hostname and path.