AI bots are blowing past robots.txt rules

For thirty years, the entire web ran on an unspoken honor system.

You drop a simple plain text file called robots.txt in your root directory, list out the folders scrapers shouldn't touch, and search engines politely turn around at the door.

Now, AI companies are treating that door like it doesn't exist.

Walking past the velvet rope

According to a report from Reuters, content licensing startup TollBit sent a letter warning publishers that multiple AI companies are actively bypassing the robots.txt protocol to scrape site content without permission.

TollBit noted that this is not isolated behavior from a rogue bot. The company tracks web traffic logs across partner publishers, and as more server logs come in, a widespread pattern of multiple AI agents ignoring exclusion rules is emerging.

The friction is already spilling into public view. Forbes recently called out AI search engine Perplexity, accusing the platform of ripping off its proprietary investigative reporting for AI-generated summaries without proper attribution or consent. Perplexity didn't comment on the claims.

The robots.txt standard was created back in the mid-1990s. Its original goal was modest: prevent aggressive web crawlers from crashing small servers. It carries zero legal enforcement. For decades, tech companies respected it simply because playing nice kept the ecosystem alive.

Generative AI seems to have thrown that courtesy out the window.

Caught between a rock and a scrape

Publishers are finding themselves in an impossible position.

Take Google, for example. If an outlet attempts to block Google's AI models from digesting their articles for AI search summaries, they often risk losing visibility in standard search rankings. That means kissing their primary web traffic goodbye.

Worse, if scrapers simply ignore robots.txt directives anyway, publishers take the visibility hit for nothing while AI companies still walk away with their words.

It is worth noting that TollBit is not an unbiased bystander here. The startup acts as a tollbooth between publishers and AI labs, offering analytics and trying to broker paid licensing deals. TollBit claimed to have 50 publisher sites on its platform as of May, though it kept the names private.

On the other side, many AI developers maintain that reading publicly accessible data off the open web violates no existing laws, as long as it is not behind a paywall.

Why it matters

We think this signals the official death of the gentleman's agreement that built the open web.

For decades, the deal was fair: search bots crawl your writing, and in exchange, they send curious humans with clicks to your website. Generative AI fundamentally breaks that loop by providing direct answers, and now it looks like several AI firms are taking the source material even after being explicitly told no.

If ethical standards cannot stop crawler agents, publishers will have no choice but to deploy aggressive IP blocking, Cloudflare challenges, and hard login walls.

In our view, AI labs aggressively dodging robots.txt are scoring cheap data in the short term, but shooting themselves in the foot for the future. Starving the open web of revenue and forcing quality writing behind walled gardens will ultimately destroy the very training data frontier models need to survive.