This started as a comment
AI-crawler response codes. Source: Cloudflare Radar, AI Insights, 7-day view, 18 Jul 2026.
Read that again. More than one in fifteen requests from AI crawlers is being actively refused at the edge — 403 or 429 — before it ever touches the content. Not deprioritised. Refused.
And most of that isn't a decision. It's a default — a managed WAF ruleset, a "block AI bots" toggle flipped in 2024, a bot-fight score set too high. The same dashboard's robots.txt tracker shows GPTBot, CCBot and ClaudeBot as the most-disallowed crawlers across the top domains — much of it inherited, not authored.
AI bot transparency tracking. Source: Cloudflare Radar, AI Insights, as of 18 Jul 2026.
So real verification isn't the name. It's the origin. Two ways:
1. Verify by published IP ranges. OpenAI, Anthropic and the rest publish their crawler IPs — openai.com/gptbot.json, openai.com/searchbot.json, openai.com/chatgpt-user.json, and their equivalents. Reverse-DNS the requesting IP, confirm the hostname belongs to the vendor, then forward-resolve it back to the same IP. If either step fails, it's a spoofer wearing the name.
2. Verify by cryptographic signature. This is where it's heading. Web Bot Auth — a Cloudflare-led IETF draft (draft-meunier-web-bot-auth-architecture, built on RFC 9421) — has crawlers sign each request with an Ed25519 key, so identity is proven, not claimed. It's integrated into Cloudflare's Verified Bots program with a dedicated Signed Agents directory, and ChatGPT's agent was in the first signed cohort in 2025. It isn't only Cloudflare, either — AWS WAF added Web Bot Auth support in late 2025, auto-allowing verified agents.
Note the status honestly: Web Bot Auth is an active IETF draft, not a ratified standard. But with Cloudflare, OpenAI, Anthropic and AWS moving in lockstep, it's already the de-facto direction.
The point for you: "block the bad bots" and "let the AI in" are the same problem, and the User-Agent solves neither. Verify by origin or by signature — otherwise you block the crawler you want and admit the one you don't.
What to actually do
Check the logs, not the browser. Pull a week of access logs, filter to the AI User-Agents, look at the status codes. That's the whole audit — an afternoon, not a project.
If you see 403/429, find the rule. It's almost always a managed ruleset, a "block AI bots" setting, or an over-eager bot-fight score — not a line you wrote. Allowlist the crawlers you want by verified IP or Web Bot Auth, never by User-Agent alone.
The rule I took away
Your HTML is the last thing in the request's path, not the first. Before a bot reads a single byte of it, the request has to clear DNS, the CDN, the WAF, the rate limiter, the bot-management score. Any one of them can end the request with a status code you never see — on a page that renders flawlessly for you.
"It works in my browser" was always a weak claim. For crawlers it isn't even the right question. The right question is: what status code did the bot get? And the only honest answer is in the logs.
Related reading:
SOCIAL SHARE CARD GENERATOR