A new web standard forces AI bots to pay up

For the past two years, web publishers facing AI scrapers had two bad choices: block crawlers completely via robots.txt and vanish from generative search, or hope they were big enough to cut a private multimillion-dollar licensing deal with OpenAI.

Now, the creators behind RSS are trying to build a third path that actually makes AI pay per click.

On Wednesday, major web publishers including Reddit, Yahoo, Medium, Quora, and wikiHow threw their weight behind Really Simple Licensing, or RSL. It is an open content standard designed to let any website set machine-readable pricing and licensing terms for AI scrapers.

Licensing terms baked into robots.txt

The standard is led by the RSL Collective, helmed by RSS co-creator Eckart Walther and former Ask.com CEO Doug Leeds. Instead of inventing a completely new protocol, RSL builds directly on top of the decades-old robots.txt file.

Instead of a blunt yes-or-no switch for specific bots, publishers can now embed royalty rules and licensing options directly into their site metadata, eBooks, or datasets.

The pricing tiers are flexible. Sites can offer subscriptions, charge a pay-per-crawl fee every time an AI bot scrapes a page, or even charge a pay-per-inference fee when a model uses their content to answer a prompt. Traditional bots crawling for search indexing or web archives are left untouched.

To give the standard actual teeth, the collective is partnering with CDN provider Fastly. Think of Fastly as the club bouncer checking IDs at the door, while RSL issues the tickets. AI crawlers that agreed to the license get in, while unauthorized bots get blocked at the edge. Publishers not using Fastly, however, will have to wait for other infrastructure providers to adopt the tooling.

The ASCAP playbook for the AI era

Beyond the tech, the collective is operating like a music rights organization such as ASCAP. By banding together, member publishers share the legal costs of taking non-compliant AI companies to court.

Leeds noted that bringing major platforms together creates collective bargaining power. AI developers can either negotiate with everyone at once, or violate everyone at once.

Currently, publishers like O'Reilly, People Inc., and Ziff Davis have also signed on, and joining the collective is free for creators.

Why it matters

We think piggybacking on robots.txt and CDNs is a clever move. It avoids the friction of new proprietary protocols and brings standardized licensing within reach of smaller sites that cannot negotiate bespoke enterprise deals with Big Tech.

For creators and mid-tier publishers, this is worth tracking closely. If widely supported, it could turn unauthorized scraping from an existential threat into an automated revenue stream.

However, the standard faces massive practical hurdles. Pay-per-inference sounds great on paper, but verifying when an AI model secretly references proprietary data inside a black-box prompt is nearly impossible without full cooperation from model builders.

More importantly, AI giants have a well-documented habit of ignoring robots.txt altogether while claiming fair use in court. Unless the RSL Collective can secure major legal precedents or get cloud giants like Cloudflare and AWS fully on board as gatekeepers, AI companies may simply choose to route around the bouncer.