Cloudflare launches a one-click block for AI scrapers

Content creators have spent the last two years watching multi-billion-dollar AI companies vacuum up their websites to train models, often without permission, credit, or compensation. If you were big like Reddit, you could ink a reported $60 million annual licensing deal with Google. If you were a normal site owner, you basically just had to sit there and take it.

That dynamic just took a massive hit. On July 3, Cloudflare rolled out a free, one-click button that lets any website owner completely block AI bots, scrapers, and crawlers. It is available immediately to all customers, right down to the free tier.

To turn it on, site owners just have to hop into the Security section of their Cloudflare dashboard, head to Bots, and toggle on the switch labeled AI Scrapers and Crawlers.

ByteDance and OpenAI are dominating the crawl

Alongside the release, Cloudflare pulled the curtain back on just how aggressively these models are hitting the open web. Looking at traffic across its massive global network, four bots dominate the charts: ByteDance's Bytespider, Amazonbot, Anthropic's ClaudeBot, and OpenAI's GPTBot.

ByteDance’s Bytespider took the crown by a wide margin. It racked up the highest request volume, reached the most properties, and was accessed by 40.40% of all websites protected by Cloudflare. OpenAI’s GPTBot came in second at 35.46%, followed by ClaudeBot at 11.17%.

What is striking is how defenseless most of the web still is. In June, AI bots accessed nearly 39% of the top one million websites using Cloudflare, yet only 2.98% of those sites actively took steps to block or challenge them.

Part of that comes down to awareness. Cloudflare’s Radar team found that while admins frequently add GPTBot or CCBot to their robots.txt files, they almost never remember to block aggressive newcomers like Bytespider or ClaudeBot.

Spoofing user agents will not save them

Even when webmasters try to protect themselves, robots.txt is completely toothless. It relies on the honor system. If a scraper decides to lie about its user agent and pretend to be Chrome on a Mac, standard rules simply fail. Just recently, search startup Perplexity faced backlash over claims it impersonated legitimate human visitors to scrape content.

Cloudflare is tackling this by taking user agents out of the equation entirely. Processing an average of over 57 million requests per second, its global machine learning models fingerprint automated behavior at the network level.

In Cloudflare's scoring model, anything under 30 is flagged as a bot. When analyzing evasive scrapers that intentionally disguise their identity, the system scored the vast majority at 9 or below. In other words: lie all you want in your header, the underlying traffic fingerprint still gives you away.

Why it matters

This is a massive power shift back toward creators and publishers. Up until now, defending your content against AI ingestion required sophisticated Web Application Firewall rules, expensive enterprise software, or constant manual updates to blocklists. Cloudflare giving this away to free accounts levels the playing field instantly.

For the AI sector, the era of treating the public internet as an endless, free training buffet is rapidly closing. Cloudflare sits in front of a staggering percentage of the web. As more publishers toggle this switch, high-quality human text will become significantly scarcer, forcing foundation model builders to either pay up for data or rely on messy synthetic content.

We think Cloudflare played this brilliantly. By catering to publisher outrage, they reinforce their position as the default protector of the open web while making their free tier even stickier. The only catch is the inevitable escalation: scraper teams will almost certainly burn more capital on sophisticated residential proxies to mimic human browsing, kicking off an even nastier cat-and-mouse game.