Everyone's writing think-pieces about "optimizing for AI search." I wanted numbers. So I pulled together the 2025-2026 data from Cloudflare, Vercel/MERJ, Ahrefs and the AI providers' own docs. Some of it genuinely changed how I build sites. Here's the reality check, dev to dev.
1. The AI crawlers do not run your JavaScript
This is the big one. A Vercel/MERJ analysis of 500M+ crawler fetches found that none of the major dedicated AI crawlers execute JavaScript — not OpenAI's GPTBot/OAI-SearchBot/ChatGPT-User, not Anthropic's ClaudeBot, not PerplexityBot, not Meta or ByteDance's Bytespider. They fetch your raw HTML and leave. They do download JS files (~11.5% of ChatGPT's fetches, ~23.8% of Claude's) — they just never run them.
The exception is Google: its Web Rendering Service is shared across Search and Gemini, so Google-Extended renders JS the same way Googlebot does.
Translation: if your content only exists after client-side hydration, every AI engine except Google sees an empty shell.
# see what the bots see
curl -A "GPTBot" https://yoursite.com/page | grep "your headline"
If that's empty, you need SSR, SSG, or prerendering.
2. AI crawl traffic is exploding — and barely sends anything back
Cloudflare's data shows AI crawler activity up sharply year over year, with GPTBot's volume up ~305% and reaching ~11.7% of all crawler requests by mid-2025. But roughly 86% of that crawling is for model training, not live search retrieval.
And the referrals? Lopsided doesn't cover it. Measured crawl-to-referral ratios hit ~38,065:1 for Anthropic and ~1,091:1 for OpenAI. They read enormously, cite rarely. AI tools still send under 1% of outbound web traffic today.
3. The SERP is going zero-click
By early 2026, roughly 68% of Google searches ended without a click, and pages with an AI Overview above them saw click-through drop by around 60%. Being "on page one" is worth less every quarter; being the cited source is worth more.
4. Two overhyped tactics, with the data attached
llms.txt: adoption grew ~8.8x, but in one May 2026 measurement 97% of llms.txt files received zero requests. OpenAI, Anthropic and Google all point site owners torobots.txtinstead. Add it if you like — it's a cheap static file — but don't expect it to do the work.
Schema/JSON-LD as a citation hack: controlled analysis found schema presence does not independently predict AI citation once you correct for content quality. Schema still earns rich results and disambiguates entities — just don't treat it as a magic AI-ranking lever.
What actually moves the needle
Boring, durable engineering:
Server-render your content — verify withcurl, not DevTools.
Allow the retrieval bots you want cited by inrobots.txt; block only what you must.
Answer-first, semantic HTML — clean headings, real lists, a direct answer per section.
Core Web Vitals, INP included — fast pages get crawled more thoroughly.
None of this is a hack. It's the same discipline as good, crawlable web engineering — which is why developers, not marketers, tend to be the ones who move AI visibility. It's the technical backbone of modern .
Build for the crawler that can't run your JavaScript, and you're ready for whatever the answer engines do next.
Sources: Cloudflare Radar, Vercel/MERJ crawler study, Ahrefs brand-mentions analysis, and OpenAI/Anthropic/Google bot docs (2025-2026).
SOCIAL SHARE CARD GENERATOR