Cloudflare lets websites block or charge AI crawlers

For years, AI labs have freely scraped the open web while publishers watched their referral traffic dry up. Now, the bouncer of the internet is putting up a tollbooth.

Cloudflare announced on Tuesday that it is rolling out a feature that lets websites either block AI chatbots entirely or charge them a flat, per-request fee to crawl content.

Matthew Prince, co-founder and CEO of Cloudflare, put it bluntly: the web is being stripmined by AI crawlers, with creators seeing almost no traffic and therefore almost no value.

Three switches and a paywall

The new tool gives website owners three clear options.

You can keep the doors open and allow AI bots free access, slam the door shut and block them all, or charge a uniform, per-request price across your entire site.

If you want to play nice with certain partners, Cloudflare also lets you bypass charges for specific crawlers, making it easier to honor private licensing deals.

Cloudflare framed the rollout as a technical shift in how digital content is controlled, designed to hand creators leverage over who accesses their work.

Scrapers up 65 percent

This move comes after months of escalating crawler pressure.

According to Cloudflare, AI training crawl activity jumped 65% over the past six months alone.

Last July, the company rolled out a simple one-click blocker for AI bots, and more than one million site operators turned it on.

With Cloudflare estimating that around 16% of global web traffic flows through its infrastructure, this is not a niche experiment.

Prince argued that content is the fuel powering AI engines, so it is only fair that creators receive direct compensation.

Why it matters

For independent sites and media brands, this is a massive shift in power.

Small publishers never had the clout to strike multi-million-dollar data licensing deals like Reddit or Axel Springer. A standardized pay-per-crawl feature turns every small blog into a paid data vendor overnight.

We think Cloudflare is pulling off a brilliant platform play here by turning its security proxy into the de facto clearinghouse for web scraping.

However, it also carries real friction: well-funded labs like OpenAI and Google can afford the toll, but independent AI researchers and smaller startups will find the open web closing in fast.

The era of free training data looks effectively over, and the internet is quietly entering its pay-per-token reality.