Is Your Site Blocking AI Crawlers? Audit robots.txt, Your CDN, and Your Renderer

You publish an answer-first article, add schema, submit the sitemap, and wait. Three months later ChatGPT still names a competitor. The usual response is to blame the content and rewrite it. Sometimes that is right. But before you rewrite anything, it is worth spending forty minutes proving that AI crawlers can actually fetch the page, because a surprising number of well-run sites are quietly returning 403s to GPTBot while serving Googlebot a clean 200.

Nearly every guide on this topic tells you to open yoursite.com/robots.txt and look for a Disallow line. That check is necessary and it is nowhere near sufficient. Robots.txt is a request that a well-behaved crawler chooses to honor. It is not the thing that stops a crawler. The things that actually stop a crawler are your edge network and your rendering strategy, and neither of them shows up in a text file.

Here is the full audit, layer by layer, with the exact commands.

The three layers where AI crawlers die

Think of a crawler request as passing three gates before it becomes indexable text.

Most teams check layer one, find nothing, and conclude they are fine. Layers two and three are where the interesting failures live, and they are invisible from the browser because your browser is not sending a bot user agent.

Layer 1: read robots.txt correctly, not just quickly

Start with the file, but read it the way a crawler parses it rather than the way a human skims it.

Two parsing rules cause most of the mistakes.

The most specific user-agent group wins, and only that group applies. If your file has a block for User-agent: * and a separate block for User-agent: GPTBot, GPTBot obeys only the GPTBot block. It ignores the wildcard block entirely, including any Allow rules you put there. Teams get bitten when they add a narrow GPTBot group for one purpose and accidentally strip away the general permissions.

An allowlist pattern silently blocks everyone you did not name. This is the single most common accidental block. A file that reads:

User-agent: Googlebot
Allow: /

User-agent: *
Disallow: /

is a complete block on every AI crawler in existence. It looks like a permissive file. It is a deny-by-default file with one exception. If your robots.txt was written by a security-minded engineer or generated by a plugin with a strict preset, check for this shape first.

Then check the user agent list itself. The names split into three jobs, and the distinction matters because blocking the wrong one costs you differently.

The practical takeaway is that the retrieval crawlers are the ones that move citations this quarter. If you are going to be selective, block training crawlers and keep the search crawlers open. Blocking OAI-SearchBot and Claude-SearchBot while leaving GPTBot open is close to the worst possible configuration, and it happens when someone copies a partial list from a blog post.

Operator documentation is the authority on which names are current. OpenAI publishes its bot names and IP ranges in the OpenAI docs, Anthropic documents how site owners can control Claude's crawler, Google lists Google-Extended among its common crawlers, and Perplexity documents PerplexityBot and Perplexity-User. Check these rather than a listicle, because the names change and the listicles do not.

Layer 2: the edge, where the real blocks are

This is the layer almost no one audits, and it is the one that produces the confusing symptom: robots.txt is clean, the page is fine in a browser, and the crawler still never sees it.

Bot management products classify traffic by signature, not just by user agent string. An AI crawler that identifies itself honestly can be caught by a managed rule for automated traffic, a rate limit tuned for scrapers, a geographic block, or a challenge page. From the crawler's side, all of those look like a 403, a 429, or an HTML interstitial with no article text in it. None of them appear in robots.txt.

Cloudflare in particular has shipped increasingly aggressive defaults here. It now offers explicit AI crawler controls that let you allow or block AI bots by category, and some plans have had blocking enabled by default at various points. If your site sits behind Cloudflare and you did not personally configure that setting, you do not know what it is set to. Go look.

Test it with a live request

The only reliable check is to make the request yourself as the bot. From any terminal:

curl -s -o /dev/null -w "%{http_code}\n" \
  -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot" \
  https://yoursite.com/your-best-article

Run it once per crawler. A 200 is a pass. A 403 is a hard block. A 429 is a rate limit, which is a partial block that gets worse as the crawler tries to work through your archive. A 503 with a short body is usually a challenge page.

Two refinements that catch subtler failures:

Compare against a browser user agent. Run the same URL with a normal Chrome user agent string. If Chrome gets 200 and GPTBot gets 403, you have found your block and you know exactly which layer it lives in.

Compare the byte count, not just the status. A challenge page returns 200 in some configurations. Add %{size_download} to the curl format string. If the bot request returns 200 with 4 KB and the browser request returns 200 with 90 KB, the bot is getting an interstitial, not your article.

curl -s -o /dev/null -w "%{http_code} %{size_download}\n" -A "<agent>" https://yoursite.com/page

Then confirm in your logs

A curl test proves your edge will serve a request that claims to be GPTBot. It does not prove that the real GPTBot came and got the page. For that, grep your server or CDN logs for the user agent string over the last 30 days.

What you are looking for is three things: that the crawler appears at all, that its status codes are 200 and not 403, and that it is reaching your content URLs rather than only the homepage. A crawler that fetches your homepage and nothing else usually means it hit a block or a rate limit partway through and backed off.

If your logs show requests claiming to be GPTBot from IP addresses outside OpenAI's published ranges, those are impostors, which is common and mostly harmless noise. Verify by checking the source IP against the operator's published ranges before you treat a log line as evidence of a real crawl.

Layer 3: the renderer, where the page arrives empty

The third block is not a block at all. The crawler gets a 200, gets a real page, and finds nothing worth quoting because your content is assembled in the browser.

AI crawlers are, as a class, far less willing to execute JavaScript than Googlebot is. Googlebot has spent a decade building a rendering pipeline. The retrieval crawlers feeding AI answers are optimized for speed and volume, and most of them read the HTML they are handed. If your article body is injected client-side by React, Vue, or a headless CMS widget, the crawler sees your nav, your footer, and a loading state.

The test takes one command:

curl -s https://yoursite.com/your-best-article | grep -c "a distinctive sentence from your article"

If that returns 0, your content is not in the initial HTML. Also try piping the raw response to wc -c and comparing it to what you would expect for a 2,000-word article. A 12 KB response for a long article is a shell.

The fix is architectural: server-side rendering, static generation, or a prerender layer. This is not an AEO-specific ask. It is the same fix you would make for any crawler. But it is worth checking before you conclude that your content structure is the problem, because no amount of answer capsule tuning helps a page that arrives as an empty div.

One more renderer-adjacent failure: content behind a cookie banner or consent wall that blocks the body until interaction. Crawlers do not click accept. If your consent implementation hides the article rather than just the tracking, the crawler gets nothing.

The 40-minute audit, in order

Run these in sequence. Stop when you find a failure, fix it, then continue.

1. Fetch robots.txt. Look for the allowlist-only pattern first, then for named AI bots. Confirm which group each crawler would match.
2. Curl your top ten pages as GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, and PerplexityBot. Record status code and download size for each.
3. Curl the same pages as Chrome. Diff the results. Any gap is an edge block.
4. Open your CDN's bot management panel. Read the AI bot setting explicitly rather than assuming.
5. Grep 30 days of logs for each crawler name. Confirm presence, status codes, and URL depth.
6. Grep the raw HTML of a content page for a sentence from the body.
7. Check your sitemap and llms.txt are reachable with a bot user agent too. A blocked discovery file means the crawler never learns your URLs exist.

That last point is where most teams' AI feed work quietly fails. Publishing an llms.txt does nothing if the file itself sits behind the same bot rule blocking everything else. If you have not built one yet, our free llms.txt generator will produce a valid file you can drop at your root, and our guide to llms.txt and schema for AI crawlers covers what those files do and do not accomplish once they are reachable.

What to do once access is clean

Fixing a block does not produce a citation. It produces eligibility. The crawler now has to come back, recrawl, and the engine has to decide your page is the best source for a query.

Expect the sequence to run in that order and on that timescale: access fix, then recrawl over days to weeks, then citation changes after that. Retrieval crawlers move fastest because their indexes refresh continuously. Training crawlers move on model release cycles, which means a block you lift today may not show up in baseline model knowledge for a long time.

This is also the point where an access fix and a content problem become distinguishable. If you unblock, confirm crawls in your logs, wait a full recrawl cycle, and still see zero citations, the problem was never access. It is that your page is not the best answer for the query, which is a content and entity problem rather than an infrastructure one. Our breakdown of why a page can rank on Google and still be invisible in ChatGPT covers what changes at that point.

Keeping a standing feed of clean, crawlable, answer-shaped content is the durable version of this. That is what the AI Feed Engine does, and how OnlyAEO works walks through the measurement side, from crawl confirmation through citation share. The FastTrackr AI case study shows what the sequence looks like end to end for a brand starting from zero visibility.

Make it a monitor, not a one-time check

Access is not a state you achieve. It is a state that regresses. A CDN plan change, a new security plugin, a WAF rule someone added during an incident, a framework upgrade that moved rendering client-side: any of these can reintroduce a block months after you cleared it, and nothing will alert you.

Set a recurring check. The cheap version is a weekly cron job that curls five URLs with five bot user agents and posts the status codes to a Slack channel. The better version watches your logs for crawler presence and alerts when a crawler that used to appear goes quiet for a week, because a rate limit or a partial block shows up as a decline in crawl volume long before it shows up as a decline in citations.