Cloudflare gives AI crawlers an ultimatum: split search from training or get blocked

If you run a website, you know the frustration: you want Google to index your pages for search traffic, but that same crawler scoops up your articles to train AI models that answer user questions directly, starving you of clicks and ad views.

Now, Cloudflare is stepping in as an enforcer. The networking giant just gave AI companies an ultimatum: separate web crawlers used for traditional search from those used for AI training and agents by September 15, 2026.

Once that deadline hits, Cloudflare will automatically block mixed-use crawlers by default on any pages running ads. The change will apply automatically to all new customers, any new sites created by existing users, and every current free-tier account.

Forcing Google to untangle its bots

Most site owners are not completely anti-AI. They want their work to be discoverable, but they are sick of giving away their intellectual property for nothing.

Cloudflare specifically called out the world's largest search engine, noting that Google effectively accesses twice as much web data as rival AI developers because it makes opting out of AI without losing search traffic nearly impossible.

Google has pushed back on this characterization before, pointing to its Google-Extended token that lets publishers opt out of training systems like Gemini and Vertex API without losing standard search placement. Yet the main Googlebot still scrapes content directly for AI Overviews and AI Mode features within search results.

Cloudflare CEO Matthew Prince explained that action could not wait because automated bot traffic recently surpassed human web traffic for the first time in history, a milestone analysts did not expect until next year. If the web is mostly bots now, the rules of engagement have to change.

It is also a waste of infrastructure. Cloudflare data revealed that over 50% of all AI crawler activity is spent pointlessly re-indexing pages that have not changed at all.

Turning scraping into a commercial transaction

Alongside the deadline, Cloudflare is broadening its publisher monetization tools. The company is evolving its Pay Per Crawl marketplace into a broader Pay Per Use program, letting publishers get paid when their content actually generates commercial value rather than just when a server downloads an HTML file.

Initial rollout partners include Ceramic.ai and You.com. When publishers opt in, they will receive payments when their writing shows up in Ceramic's AI answers or when You.com accesses premium articles.

Why it matters

We think Cloudflare is pulling off a brilliant tactical play. By sitting in front of a massive portion of the world's internet traffic, it is effectively appointing itself as the union rep and toll booth operator for publishers against AI labs.

For creators and publishers, this is long overdue leverage. Historically, opting out of AI scrapers meant risking total disappearance from search engines. Cloudflare turning mixed-use crawler blocking on by default for ad-supported sites suddenly shifts the burden back onto big tech companies.

That said, there is an obvious catch with Pay Per Use: auditing. Measuring exactly when and how a piece of text creates value inside a complex generative answer is notoriously difficult. If the payouts end up being pennies while legal liability and compute remain hazy, publishers might just block agents altogether.

For the AI ecosystem, the clock is ticking. Companies can no longer hide AI scrapers behind friendly search crawler user-agents. The era of free, unmetered web scraping is rapidly closing.