You've spent months optimizing your content for ChatGPT citations. Structuring your data, front-loading the substance into the first third of every page, following every GEO tip you've come across. Now Cloudflare announces that starting September 15 it will block AI crawlers by default on sites carrying advertising. The infrastructure handling 20% of web traffic is padlocking the very door you'd spent months leaving wide open.
TL;DR: The no-fluff summary
- The block: Cloudflare cuts off training and agent AI crawlers by default on ad-supported sites. Traditional search crawlers still get through.
- The GEO paradox: 44.2% of ChatGPT citations come from the first 30% of your content, but the crawler that needs to read it is the same one now being blocked.
- Pay Per Use: Cloudflare evolves its model so you get paid when your content appears in an AI response. Sounds reasonable, but the numbers aren't worth celebrating.
- The ratios that sting: 73,000 pages crawled per visit returned (Anthropic, 2025). The old content-for-traffic deal was already dead.
What AI Crawlers Is Cloudflare Blocking Starting September?
Cloudflare will activate by default the blocking of "mixed-use" AI crawlers on ad-supported sites starting September 15, 2026. The change affects new customers, new sites, and free plan users. Traditional search crawlers still get through. Training and agent crawlers are out.
The nuance: Cloudflare has asked operators like Googlebot, Applebot, and BingBot to separate their functions. Crawling to index for a search engine is one thing; crawling to feed an AI model is another. If they don't comply before September, the block will catch them in full.
The numbers speak for themselves. According to Cloudflare's own data, crawl-to-traffic ratios in 2025 were 14:1 for Google, 1,700:1 for OpenAI, and 73,000:1 for Anthropic. SEVENTY-THREE THOUSAND pages read for every visit they send back. The old deal, "let me read your site and I'll send you users", was already dead. Cloudflare just filed the death certificate.
The GEO Paradox: Optimizing for Someone Who Doesn't Pay
Here's where it gets uncomfortable.
A study by Kevin Indig published in Search Engine Land found that 44.2% of ChatGPT citations come from the first 30% of a page's content. Not from the bottom, not from a hidden corner, from the top. That finding has fueled an entire wave of consultants selling "GEO strategy": structure your content so AI models cite you, as we analyzed in our guide on GEO and the keyword-stuffing trap.
The problem: the crawler that needs to read that first third to cite you is exactly the one Cloudflare is about to block. The pipeline your AI visibility travels through is the same one draining your content without compensation.
Worth noting: a large share of AI citations don't even overlap with Google's organic results. They're separate channels. You can rank on page one and not exist for ChatGPT. Optimizing for AI citations was already a separate discipline from classic SEO. Now there's a toll booth in the middle of the road.
In my experience, doing GEO without a clear plan for monetizing your crawl exposure means giving away your content in exchange for a mention you can't properly measure. It's like letting someone photocopy your entire book on the understanding that they might recommend it in conversation. That's a bad deal, because it is one.
Pay Per Use: Fair Compensation or Just Another Toll?
Cloudflare has shut the door, yes. But it's also made a move: it's evolving Pay Per Crawl (launched in 2025) into a Pay Per Use model. The difference matters. You're no longer paid just for letting someone crawl your site, but when your content actually appears in a chatbot's response. The technical mechanism is an HTTP 402 ("Payment Required") status code, with Cloudflare managing billing. First partners on board: Ceramic.AI and You.com.
On paper, it sounds reasonable. You create content, a chatbot uses it, you get paid.
The uncomfortable question: who's actually going to pay?
OpenAI and Google have the resources to negotiate. What about smaller AI labs? If they can't absorb the cost of fresh data, the model ends up concentrating information in the hands of the biggest players, the same ones that already dominate. Pay Per Use may end up protecting incumbents long before it compensates any publishers.
Most internet traffic is already non-human. Cloudflare's own data confirms it. And I'd bet that within six months, half the industry will be selling "Pay Per Use strategies" as the next gold rush, without anyone stopping to show the actual numbers. If the ratio is 73,000 crawls per visit Anthropic sends back, imagine the fraction of a cent that works out to per generated citation. Do the math.
This Isn't About a Toggle. It's About Your Business Model.
Block or don't block? Charge or give it away? Everyone reduces this to a button in a control panel. What you're actually deciding is how you make money from your content.
Generic content, the same tutorial that exists on 400 other sites, nobody is going to pay to crawl it. And blocking it won't save you either; it just makes you invisible in two channels instead of one.
It's a different story if you have proprietary data, original analysis, research that doesn't exist anywhere else. That's where the leverage shifts. As we analyzed in the guide on agentic search and SEO's renewed role, brand and content quality carry more weight than ever for visibility in both search engines and AI models. Cloudflare's move fits the same pattern: if your content doesn't offer something different, it gets left out.
Every content lead should be asking themselves one question: "Is my content valuable enough that someone would actually pay to access it?" If that question makes you uncomfortable, the problem isn't Cloudflare.
Cloudflare's AI crawler block is going to force the entire industry to put a price on something it's been giving away for years. Those already losing organic traffic will have to decide whether to open a new revenue stream or watch AI models consume their best content without picking up the tab. The numbers are on the table. Read them before you buy the next pitch.
Frequently Asked Questions About the AI Crawler Block
What's the difference between Pay Per Crawl and Pay Per Use?
Pay Per Crawl compensates publishers each time an AI bot crawls their site. Pay Per Use goes further: payment is triggered only when the crawled content actually appears in a chatbot's response. Cloudflare has moved from the first model to the second so that compensation is tied to the real value your content delivers, not just to a bot having visited.
Does Cloudflare's block affect Google search rankings?
Not directly. Cloudflare is blocking AI training crawlers and agents, but traditional search crawlers (like Googlebot for indexing) still get through. Organic search rankings are unaffected. What changes is your visibility in generative engines like ChatGPT or Perplexity, which depend on the crawlers now being blocked.
