You keep an origin healthy while staying open to AI agents by caching the cheap, near-static surfaces agents read most and reserving your compute for the offer data that has to be fresh, not by blocking the agents that would have selected you.
Key takeaways
- A live, user-triggered agent fetch hits your origin at question time while a shopper waits, so the cost of being open is real, but the answer is caching, not blocking.
- Retail is one of the sectors AI crawlers hit hardest, and crawl-to-refer ratios can be extremely lopsided, so plan for the fetch load instead of being surprised by the bill.
- Edge-cache the near-static machine surfaces (llms.txt, Markdown mirrors, feeds, mostly-static product pages) and keep price and availability on a short leash so an agent never quotes a stale offer.
- Access control (throttling unverified spoofers) is not content variation (serving different content by user-agent, which is cloaking); allowlist the verified agents and rate-limit only the fakes.
This chapter is about affording openness, not blocking agents. Every load figure below is a reason to cache smarter, never a reason to slam the door on an agent that would have selected you. Rate-limiting here targets unverified spoofers only. The verified shopping and answer agents should always get through.
Why agent traffic strains an origin differently
A scheduled search crawler and a live shopping agent load your server on opposite economics. A classic index crawler visits on its own timetable, so its cost is smooth and predictable: fetch once, and the engine can send you many shoppers off that one visit. A live agent fetch is the reverse. It fires the moment a real person asks a question, it is tied to that person waiting for an answer, and nothing guarantees the work sends a visitor back.
That second mode is growing fast. Cloudflare recorded "user action" crawling, meaning bots that simulate a human action rather than run a scheduled index, growing more than 15x across 2025, and more than 21x from the start of January to early December. (, 2025-12-15) Read that as a range across engines, not a fixed toll: the point for a merchant is that "being crawled" and "being sent shoppers" are not the same size, so you should budget for the fetch load on its own terms.
Retail sits squarely in the blast radius. Cloudflare reported that retail and computer software were consistently the two sectors that drew the most AI crawler traffic, together just over 40% of all AI crawler activity it observed. (, 2024-07-24) Read the Docs reported a single crawler downloading 73 TB of zipped HTML in one month (roughly 10 TB of it in a single day) at a bandwidth cost of over $5,000; after it blocked the abusive crawlers, downloaded-file bandwidth fell about 75% (from roughly 800 GB/day to roughly 200 GB/day), which it estimated at about $50 a day, or $1,500 a month, in avoided cost. ( user-agent token belongs to which mode is a table we already maintain, so we will not restate it here. See the , any , which is the other half of this decision: pack them with verifiable facts, then make them cheap to serve.
Expensive, and not safely cacheable for long, is the live offer data: current price, stock status, and the delivery window for this shopper. This is the information that changes often and that an agent most needs to have correct.
The trap is the long tail. Wikimedia reported that at least 65% of its most resource-consuming traffic came from bots even though bots were only about 35% of pageviews, because crawlers bulk-read many low-popularity pages that human readers rarely open and that therefore miss the cache. (, 2025-04-01) The lesson for a store: the requests that hurt are usually not the homepage, which is always warm in cache, but the thousands of deep product and variant URLs a crawler will read once and a shopper almost never will. Cache those on purpose, or they find your origin every time.
Edge-caching the machine surfaces
This is the part you hand to a developer, and it is deliberately short, because the caching directives themselves are documented by the CDN and web-server vendors and we link straight to them. The behavior you want is simple to state: serve the cached copy instantly, and refresh it in the background. Two mechanisms give you that.
Cloudflare serves stale content while it revalidates only if your origin's Cache-Control header includes stale-while-revalidate; when that is set, the first request after the cached copy expires triggers an asynchronous background revalidation and immediately receives the stale content marked with an UPDATING status, instead of blocking on the origin. ()
These are two separate implementations, not one feature: nginx's serve-stale-while-refreshing is not identical to Cloudflare's asynchronous UPDATING flow, so test the one you actually run rather than assuming they behave the same. If you want the edge to hold a copy longer than a browser does, Cloudflare's cache-control guidance is to send max-age together with stale-while-revalidate from the origin and set a separate Edge Cache TTL in a Cache Rule, and it warns against combining s-maxage with stale-while-revalidate. ( has a complete stale-while-revalidate worker built on the same directives above.
The freshness tradeoff: never serve price and availability long-stale
Stale-while-revalidate is a gift for low-churn surfaces and a liability for offer data. Our stance is a hard line: cache descriptions, specs, and llms.txt generously, but keep price and availability on a short TTL, or bypass the cache for them entirely, so an agent can never read and then quote an offer you no longer honor.
The reason is selection, not correctness alone. An agent that surfaces a price you have since raised, or shows "in stock" for something you have sold out of, hands the shopper a bad checkout, and that mismatch is exactly the kind of thing an engine learns to stop trusting you for. The same freshness discipline you apply to your . Throttling a request is not content variation: rate-limiting changes how much a client can fetch, never what the page says, so it is not cloaking. Say that plainly to anyone who worries the two are the same thing.
Our recommended stance is narrow. Allowlist the verified shopping and answer agents and let them through, and reserve rate limits for traffic you cannot verify: clients that claim to be a known agent but fail that engine's published verification method. Rate-limiting the verified agents that select you would be self-defeating. How to verify each engine's bots (their published IP ranges and reverse-DNS checks) is in the , accessed 2026-07-08) That is a response to scraping abuse by projects without a budget to absorb it, and it is context here, not our advice. For a store that wants to be selected, a proof-of-work wall that challenges every browser-like client is the opposite of the goal: the aim is to make the door cheap to hold open, not to close it.
What to measure
Four numbers tell you whether you are affording openness or quietly paying for it.
- Cache-hit ratio on the machine surfaces. Measure it per surface (llms.txt, Markdown mirrors, feeds, product pages) at your CDN or in your access logs. A low hit rate on a near-static surface means you are serving from the origin what you should be serving from the edge.
- Origin egress and bandwidth, tracked before and after you add caching. This is the bill the Read the Docs and Wikimedia numbers describe, and the clearest sign your caching is working.
- Crawl-to-refer ratio per agent. Compare requests from each engine's bots against the visits or conversions you can attribute to that engine. A very lopsided ratio is not a reason to block; it tells you which agents are worth the most caching effort and which you are subsidizing.
- 403s (and any other blocks) returned to verified agents. This is the block you did not mean to ship: a WAF rule, a bot manager, or a plugin quietly rejecting an agent you wanted. Grep your access logs for these tokens and confirm they get 200s on a live product URL.
If verified agents are being blocked and your products are missing from an engine's results, start the full diagnosis in for the selection signals that decide the outcome once your origin is healthy and open.
SOCIAL SHARE CARD GENERATOR