If your agent fetches a web page and hands the answer to a user, the page's owner increasingly wants a say in that. Cloudflare, which sits in front of roughly 20 percent of all websites, announced on July 1 that it is splitting its old single AI-bot toggle into three categories: Search, Agent, and Training. The controls went live immediately for every customer, including free tiers. The real shift comes on September 15, when the defaults flip: Agent and Training crawlers will be blocked on ad-supported pages by default, while Search stays allowed. The rule applies to new domains, existing free-tier customers, and any new site an existing customer creates. Anyone who wants different settings can opt out before that date through their security settings, though most coverage of the announcement skipped that free-tier auto-enrollment detail entirely. If you build agents that browse the open web, this is the first time a major infrastructure layer is pricing your access at zero unless you negotiate otherwise.
Cloudflare is turning the open web from an all-you-can-read buffet into a menu with prices.
What exactly did Cloudflare change on July 1?
The company replaced its single block-all-AI-bots switch with a three-way taxonomy. Search covers crawlers that index a page to answer questions about it later, the kind that send a reader back to the source. Agent covers automated systems acting in real time on behalf of a user, including ChatGPT's fetch bot and browser-driving agents that load a page to extract an answer. Training covers crawlers that pull content into a model's weights. All three toggles went live July 1 for every Cloudflare customer, from enterprise to free tier.
The September 15 default change is the part that bites. On any page that displays ads, Agent and Training will be blocked unless the site owner explicitly allows them. The logic, as Cloudflare frames it, is that an advertisement is evidence a page was built for a human to land on. A search crawler that sends a reader back is a referral. A bot that reads the page and hands the answer to someone else is not. The defaults land on ad-supported pages specifically because those are the pages agents want most: news, reviews, pricing, product coverage. Cloudflare CEO Matthew Prince said the company hopes the changes will encourage mixed-use crawlers to separate search from agent use and training, which is a polite way of saying the pressure is the point.
The Google complication is real. Googlebot crawls for both search and training in a single bot, so under the most restrictive rule, a site that blocks Training also blocks Googlebot, and your search visibility goes with it. Cloudflare knows this. The tension between blocking AI training and preserving Google search referrals is the knife edge publishers now walk.
Cloudflare says more than half of AI crawler traffic is spent re-fetching pages that have not changed, a waste figure that gives both sides a reason to price access rationally rather than simply walling everything off.
How bad is the coverage hit for agents that browse the web?
Agentic deployments have been built on the assumption that the open web stays open. A research agent fetches a competitor's pricing page. A monitoring tool checks a supplier's announcements. A customer-service agent pulls a manufacturer's specification sheet. None of this has historically required a licence, and until now, none of it needed one. Cloudflare's blocks operate at the network level, not as a robots.txt suggestion a crawler can ignore. A blocked agent does not get a lawsuit. It gets silence, or an answer built from whatever it could still reach.
The failure mode is degraded coverage, not a clean failure. The block lands on ad-supported pages and leaves the rest reachable, which means your agent might still fetch a manufacturer's spec sheet from their corporate domain but miss the third-party review on an ad-supported blog. For enterprise agents that synthesize information from multiple sources, partial coverage produces partial answers, and partial answers are wrong answers the user cannot detect.
The classification is behavioural, not something you opt into. A research agent that browses in real time is caught whether or not its operator thinks of it as a crawler. This means the surface area of risk is wider than the AI team that built the agent. If your marketing team runs a competitive intelligence tool that scrapes pricing pages, that is an Agent-class crawler under Cloudflare's taxonomy, and it will start hitting walls on September 15.
What should agent builders do before September 15?
You have a workable problem if you start before the deadline. The ones who find out from a 403 will be rebuilding on the fly.
Start by auditing which of your tools will read as Agent-class. The classification is behavioural, so you need to map every automated fetch your systems make, not just the ones your AI team owns. Marketing automation, competitive intelligence, supplier monitoring, and customer-service lookups all count.
Then expect degraded coverage and design for it. If your agent currently fetches ad-supported pages for pricing or reviews, plan for gaps. Consider which sources are essential and which are replaceable. For the essential ones, you need negotiated access, not a rewritten user-agent string. Cloudflare's Pay Per Crawl infrastructure is becoming Pay Per Use, with companies like Ceramic.ai already paying publishers when their content appears in AI search results, and You.com paying when an agent reaches premium content. The mechanism is a rate, not a wall.
What this means for you:
- Audit your fetch surface: Every automated web request your organization makes is potentially Agent-class. Map them before September 15.
- Budget for paid access: The ad-supported pages agents want most are the ones being blocked. Identify which sources are worth paying for and which you can lose.
- Build fallback paths: Degraded coverage means some sources will return 403s mid-session. Your agent needs graceful degradation, not a hard crash.
- Watch the taxonomy gaming risk: Search, Agent, and Training are behaviours the AI companies declare about their own bots. A firm that would rather not have its training run classified as training has an obvious incentive to mislabel. Cloudflare has not explained what stops this.
The deeper structural risk is the taxonomy itself. If compliance is self-declared, the companies most likely to respect the rules are the ones least likely to need enforcement against. The companies willing to mislabel their bots will get the access, and the companies playing straight will pay for it. Cloudflare needs a verification mechanism that goes beyond user-agent strings, or the taxonomy becomes an honour system with billion-dollar incentives to cheat.
What does this mean for publishers and site owners?
If you run a site on Cloudflare, check your tier first. Existing free-tier customers are moved to the new defaults automatically on September 15. If you have been relying on the old single toggle and have not touched your security settings since, your site will start blocking Agent and Training crawlers on ad-supported pages without you doing anything. That might be what you want. It might not.
The decision that matters is whether blocking Training is worth what it costs. Because Googlebot crawls for both search and training in a single bot, blocking Training also blocks Googlebot. If your search traffic matters to your business, and for most publishers it does, you need to weigh the value of keeping Google's index fresh against the value of preventing your content from being used to train competitors' models. There is no toggle that gives you both. Cloudflare is forcing the choice, and that is the point.
The money layer is where this gets interesting. Pay Per Crawl is becoming Pay Per Use, and the early deals suggest a market forming around per-access pricing rather than blanket licensing. Ceramic.ai is paying publishers when their content appears in AI search results. You.com is paying when an agent reaches premium content. These are small deals today, but they are the template. If you are a publisher, the question is whether your content is priced into these flows or whether you are giving it away while a handful of large publishers negotiate the rates.
Is the open web closing or just getting priced?
Access to the open web has been free and unlimited for roughly thirty years, and the bill is now itemised. Cloudflare's move is the first time a major infrastructure provider has made the default stance toward AI crawlers restrictive rather than permissive, and because Cloudflare sits in front of such a large share of the web, the default matters more than any individual site's robots.txt.
For agent builders, this is the end of the assumption that the web is a free API. You can still build agents that browse, but you need to plan for a world where some sources cost money, some return 403s, and the ones that stay free may change without notice. The agents that survive this shift are the ones designed for partial coverage and negotiated access from the start. The ones that assume the old web will keep working will find out the hard way that the defaults have moved.
The broader read is that Cloudflare is building the payment rails for a content market that has existed only as litigation until now. If Pay Per Use scales, the question for builders is not just how to get access but how to budget for it. An agent that fetches fifty pages per session at even a fraction of a cent per page is a different cost model than one that fetches for free. The unit economics of agentic AI just got a new line item, and the teams that model it first will be the ones that can afford to keep their agents running.
Sources
- Cloudflare AI Crawler Controls announcement
- Cloudflare Blog AI crawler controls technical details
- Cloudflare Blog Pay Per Crawl program
- AI News AI agent crawlers now need permission
- Ceramic.ai publisher payment program
- You.com agent premium content access
