Is Your Site Blocking AI Crawlers? How to Check and Fix It

A crawler can be stopped at three layers: robots.txt, the CDN or firewall in front of your site, and HTML that arrives without its text.

How do you check whether robots.txt is blocking AI crawlers?

Open https://yourdomain.com/robots.txt and, for each AI crawler, find the one group it matches and read only that group.

Google's robots.txt specification says only one group is valid for a crawler, and a named group is never combined with the * group. Under RFC 9309, the standard published in September 2022, a crawler with no group of its own obeys *, and with no * group, no rules apply.

The allowlist that blocks every AI bot

This allowlist looks permissive:

User-agent: Googlebot
Allow: /

User-agent: *
Disallow: /

No AI crawler has a group here, so each one falls to * and gets Disallow: /.

Mistakes inside named groups

• Rules don't carry over: once GPTBot has its own group, it ignores *, including the private-path lines you meant for everyone, so repeat those rules in each named group.
• Paths are case-sensitive: agent names are not, so Disallow: /Blog/ leaves /blog/ open.
• A sitemap line doesn't end a group: Google's parser ignores every line except user-agent, allow and disallow, so it reads the file below as one group, and both Googlebot and GPTBot fall under Disallow: /.

User-agent: Googlebot
Sitemap: https://www.example.com/sitemap.xml

User-agent: GPTBot
Disallow: /

Check every subdomain

Every subdomain needs its own file, because rules apply only to the host, protocol and port where the file is served. A clean file on www.example.com says nothing about docs., blog., help. or shop.

Anthropic's crawler page likewise tells site owners to set its rules per subdomain. The file must sit at /robots.txt, lowercase, at the top level, and crawlers ignore copies in subdirectories.

Check the status code the file returns

• A 4xx response: under the standard, the crawler may access anything. Google treats every 4xx except 429 that way.
• A 5xx or no response: the crawler must assume complete disallow.
• Google on a 5xx: Google stops crawling for the first 12 hours while it keeps retrying, then, if it still cannot fetch a new version, uses the last good copy for the next 30 days.

So a bot challenge that answers a crawler's request for /robots.txt with a 503 or a timeout blocks your whole site for crawlers that follow the standard, whatever the file says.

Which AI crawlers matter, and what does each one do?

The crawlers that decide whether AI answers can cite you are the search bots and user-request fetchers. Training is a separate switch, and OpenAI says each of its settings is independent of the others.

Check bot names on each operator's own page. Anthropic's current page lists three bots, and older names still found in copied files, such as Claude-Web and anthropic-ai, are not among them.

What should your robots.txt say?

Pick one of two policies: open to every crawler, or open to AI search and user-request bots with training opted out. Both keep a private area closed.

Policy A: open to all

With no named groups, every AI crawler falls to *, and the most specific rule wins, so /admin/ stays closed while Allow: / opens the rest.

User-agent: *
Disallow: /admin/
Disallow: /checkout/
Allow: /

Sitemap: https://www.example.com/sitemap.xml

Policy B: AI search yes, training no

# Everyone not named below
User-agent: *
Disallow: /admin/
Disallow: /checkout/
Allow: /

# AI training: opted out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /

# AI search and user-requested fetches: allowed, private paths repeated
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Disallow: /admin/
Disallow: /checkout/
Allow: /

Sitemap: https://www.example.com/sitemap.xml

• Private paths repeated: the search group repeats them because those bots stop reading * once they have their own group, and naming them keeps them allowed if someone later tightens *.
• Google-Extended covers grounding: disallowing it also stops Google using your content to ground Gemini answers. If Gemini matters to your buyers, move it to the allowed group.

The configuration to avoid

Avoid the reverse of Policy B: search bots blocked, training bots open. Sites opted out of OAI-SearchBot are not shown in ChatGPT search answers.

Since OpenAI treats each bot separately, blocking OAI-SearchBot while leaving GPTBot open gives up ChatGPT search and still leaves your content open to training.

Why can AI crawlers still be blocked when robots.txt is clean?

A CDN, firewall or bot management rule can refuse the request before your page is served. That rule lives in a dashboard, so it never shows up in a robots.txt audit.

Google's guidance for AI features in Search asks for both: crawling allowed in robots.txt and by any CDN or hosting infrastructure. For firewall rules, Perplexity's docs show the allow-rule pattern to copy: match the user agent and the published IP ranges together.

Check your Cloudflare AI settings

Cloudflare's AI Crawl Control, available on all plans, shows which AI services reach your content and which crawlers follow your robots.txt.

Cloudflare's documentation also says you can block AI bots by behavior with Search, Agent and Training presets.

The same page, last updated July 1, 2026, announced new defaults for new domains from September 15, 2026: Training and Agent bots blocked on pages that display ads, Search allowed.

It also said mixed-purpose crawlers combining search and training would be blocked by every training-block setting, so a training block can stop a crawler that feeds search. Check your own zone's setting.

Test the edge with a live request

Request a page with OpenAI's documented GPTBot string, then as Chrome, and compare status code and bytes:

curl -s -o /dev/null -w "%{http_code} %{size_download}\n" \
  -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" \
  https://www.example.com/your-best-page

curl -s -o /dev/null -w "%{http_code} %{size_download}\n" \
  -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" \
  https://www.example.com/your-best-page

Repeat for each bot, plus /robots.txt and your sitemap:

• 200 at Chrome's size: the bot gets what people get.
• 200 but much smaller: probably a challenge page served with a success code.
• 403: a hard block at the edge.
• 429: a rate limit, which gets worse as a crawler goes deeper.
• 503: often a challenge or an overloaded origin, and on /robots.txt it means complete disallow.

Confirm real crawls in your logs

Your logs show whether the real GPTBot came, what status codes it got and how deep it went.

Rows of 403 or 429 for OAI-SearchBot or Claude-SearchBot mean the edge is turning them away, and behind a CDN you need the edge logs too, because a request blocked at the edge never reaches your origin log.

Then verify the IP, since a request using a bot's name from outside its published ranges is not that bot:

• OpenAI: publishes IP lists for OAI-SearchBot, GPTBot and ChatGPT-User.
• Anthropic and Perplexity: publish theirs too.
• Common Crawl: runs CCBot from dedicated ranges with reverse DNS.
• Cloudflare Verified bots: meet the same test, identified through a Web Bot Auth signature, a published IP list with a stable user agent, or reverse DNS, plus obeying robots.txt.

Can a crawler get in and still find an empty page?

Yes. If your article body is assembled by JavaScript in the browser, a crawler that does not run JavaScript gets a 200 and a page with no article in it.

Google Search runs JavaScript with an evergreen version of Chromium, yet Google still recommends server-side or pre-rendering, partly because not all bots can run JavaScript. Google doesn't name those bots, so don't assume any AI crawler renders your page.

curl -s https://www.example.com/your-best-page | grep -c "a sentence copied from the page body"

A result of 0 means that sentence is not in the HTML the server sends. Fix it with server-side rendering, static generation or prerendering before you rework any content, because no amount of answer capsule work helps a page that arrives empty.

Check the consent banner too. A crawler does not click accept, so a consent wall that hides the article leaves it with the banner alone.

Do meta robots tags and X-Robots-Tag headers matter for AI answers?

For Google, yes: Google names nosnippet, data-nosnippet, max-snippet and noindex as the controls that limit what it shows from your pages in Search, AI features included.

• noindex: keeps a page out of search results.
• nosnippet: removes text snippets, AI Overviews and AI Mode included, and stops the content being a direct input for them.
• max-snippet: caps snippet length and limits how much content can be a direct input for AI Overviews and AI Mode.
• data-nosnippet: on span, div and section elements, keeps those parts out of snippets.
• X-Robots-Tag header: carries any of these rules, and it is how you apply them to PDFs, video and images.

A page must be indexed and eligible for a snippet to appear as a supporting link in AI Overviews or AI Mode, so a stray noindex or nosnippet in a template pulls every page built on it out of Google's AI answers.

When robots.txt disallows a page, Google never sees these tags. Google also says robots.txt rules for Googlebot are the control for how your site is crawled for Search, because AI is built into Search.

The other operators document robots.txt as the control. OpenAI points site owners to OAI-SearchBot in robots.txt for search opt-outs. Anthropic and Perplexity do the same for their bots.

Should you block any AI crawlers?

Block an AI crawler only for a reason you can name, and write that reason down. Good reasons include content you license or sell, a paywall, data that is your product, and competitively or legally sensitive material.

Most of those reasons are about training, and training is a separate switch from search:

• OpenAI: you can allow OAI-SearchBot while disallowing GPTBot.
• Google: Google-Extended has no effect on Google Search inclusion, but it also covers Gemini grounding, so weigh it hardest.
• Apple: pages that disallow Applebot-Extended can still appear in Apple's search results.
• Anthropic: restricting ClaudeBot signals that future material should be kept out of training, while disabling Claude-SearchBot or Claude-User may reduce your visibility in Claude's results.

Whatever you decide, robots.txt is a request. RFC 9309 says its rules are not a form of access authorization.

• User-requested fetches: OpenAI says robots.txt rules may not apply to ChatGPT-User, because a user starts those visits. Perplexity says the same of Perplexity-User.
• IP blocks: Anthropic warns that blocking its IP addresses may not work as an opt-out, because that impedes its ability to read your robots.txt.

Private content belongs behind a login.

How long until AI engines see the fix?

OpenAI says it can take about 24 hours after a robots.txt update for its search systems to adjust.

• The robots.txt cache: the standard lets crawlers rely on a cached copy for up to 24 hours unless the file is unreachable. Google generally caches the file for up to 24 hours, and may cache it longer when timeouts or 5xx errors block a refresh.
• Page recrawls: Google says crawling can take anywhere from several days to several months, depending on how often its systems decide a page needs refreshing.
• Edge changes: there is no robots.txt cache to wait out, though Perplexity notes that firewall changes can take time to propagate.

Fixing access makes your pages eligible, not cited. Whether an engine then cites you depends on answering the buyer's question better than the sources it cites now, which our breakdown of why a page can rank on Google and stay invisible in ChatGPT covers.

How we tested this on onlyaeo.com

We ran the edge test from this page against our own site on October 13, 2026. We requested our robots.txt, our sitemap.xml, a blog post and our How it works page once with a Chrome user agent and once each with the user-agent strings for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot and PerplexityBot.

• Same answer for every user agent: every request returned 200 and the same response size as the Chrome request.
• Text in the raw HTML: the blog post's paragraphs were present in the raw HTML.
• An open robots.txt: our robots.txt states an open-by-default policy and opens with a User-agent: * group set to Allow: /.

The test has a limit: it shows that our edge serves those user-agent strings to a request from our own machine. Only server logs, checked against each operator's published IP ranges, show that the real crawlers came.

What is the full AI crawler audit, in order?

Run these checks in order, fixing each failure before moving on:

1. Fetch /robots.txt on every host, record the status code, and find each AI crawler's group.
2. Request your best pages as each AI crawler and as Chrome, and compare.
3. Read your CDN or firewall's AI bot policy, including any training block.
4. Search origin and edge logs for each crawler, its status codes, and IPs inside published ranges.
5. Grep a content page's raw HTML for a sentence from its body.
6. Search templates and headers for noindex, nosnippet and max-snippet.
7. Request your sitemap and llms.txt with a bot user agent.

A sitemap or llms.txt behind the same bot rule tells the crawler nothing. If you don't have an llms.txt yet, our free llms.txt generator builds one.

Our guide to llms.txt and schema for AI crawlers explains what the file does once crawlers can reach it.

How do you keep AI crawler access from breaking again?

Make the audit a recurring check, because a CDN plan change, a new security plugin, a migration or a framework upgrade can bring a block back without anyone touching robots.txt.

• The minimum: a scheduled curl test with each bot's user agent on a short list of URLs, /robots.txt included, that alerts when a status code or response size changes.
• A better setup: also alert when a crawler that used to appear in your logs goes quiet, since a rate limit or partial block shows up there as fewer requests.

Should you fix AI crawler access in-house or with an agency?

Your own web or SEO team can run this audit if someone has a terminal, a CDN login and log access. Bring in outside help when nobody owns all of those, or when access is fixed and you are still not cited.

Ask any provider whether they:

1. Test robots.txt, the edge and rendering.
2. Confirm fixes in logs against published IP ranges.
3. Leave the training decision with you.
4. Measure your AI visibility afterwards.

OnlyAEO is an AEO and GEO agency. Our managed service includes a website health check so every AI engine can read your site.

Access is the floor. Getting cited is a content and entity question, which is where citation architecture for SaaS sites picks up.

Frequently asked questions

Does blocking GPTBot stop ChatGPT from citing my site?

No. GPTBot is OpenAI's training crawler and OAI-SearchBot feeds ChatGPT search, so disallowing GPTBot only signals that your content should not be used for training.

Will allowing AI crawlers hurt my Google rankings?

Not according to Google: Google-Extended does not affect inclusion in Google Search and is not a ranking signal, and the other AI bots are separate from Googlebot. If crawl load worries you, Anthropic supports the non-standard Crawl-delay extension, while Apple says Applebot does not follow crawl-delay.

Do I need a robots.txt on every subdomain?

Yes, wherever you want rules to apply, since a file only covers its own host. A subdomain with no file usually returns a not-found error, which crawlers read as no restrictions. That suits an open docs site, not a staging or internal host, which belongs behind a login anyway.

How do I handle an AI crawler that is not in this table?

Find the operator's own documentation for its exact user-agent token and published IP list before you write a rule, because names in copied lists go out of date. A bot with no documentation and no published ranges cannot be verified, so treat its name in your logs as a claim; Cloudflare only counts a bot as verified when it identifies itself by signature, published IP list or reverse DNS.