Anthropic's ClaudeBot bombards websites despite anti-scraping rules

If you ask Claude whether it's allowed to scrape iFixit's repair manuals to train AI models, the chatbot will politely tell you that it's strictly forbidden by the site's terms of service.

Anthropic's actual web crawler, however, didn't seem to get the memo.

iFixit CEO Kyle Wiens revealed that Anthropic's data-scraping bot, ClaudeBot, hammered iFixit's servers nearly a million times in a single 24-hour period. Beyond the strain on infrastructure, the barrage directly ignored the repair platform's explicit ban on using its content for AI training.

A million hits and a DevOps fire drill

Kyle Wiens took to X to call out the AI startup, posting screenshots where Claude itself admitted that iFixit content was off-limits. Wiens pointed out that Anthropic was not only taking content without paying, but also burning up iFixit's DevOps resources.

While iFixit is an internet heavyweight used to handling massive traffic and standard web crawlers, Wiens told The Verge that ClaudeBot's aggressive rate was an anomaly that set off internal alarms.

iFixit's terms of use explicitly state that copying content for machine learning or AI training is prohibited without prior written permission. Yet when questioned by 404 Media, Anthropic simply pointed to an FAQ page stating that its crawler only pays attention to directives inside a site's robots.txt file.

iFixit has since added a crawl-delay rule to its robots.txt file, which Wiens confirmed finally halted the onslaught. Anthropic spokesperson Jennifer Martinez told The Verge that the company respects robots.txt signals and backed off once iFixit implemented the update.

ClaudeBot's growing trail of collateral damage

iFixit is far from the only platform caught in the crosshairs. In response to Wiens, Read the Docs co-founder Eric Holscher and Freelancer.com CEO Matt Barrie both reported seeing similarly aggressive scraping from Anthropic.

This also isn't a brand-new habit for ClaudeBot. Community complaints have circulated on Reddit for months, and back in April, the Linux Mint web forum suffered an outage specifically blamed on ClaudeBot scraping.

While leaning on robots.txt is the standard opt-out approach for OpenAI and others, it leaves site owners with zero nuance to specify permissions. Other players, like Perplexity, have been caught ignoring robots.txt entirely. For now, it remains one of the few blunt tools sites like Reddit use to block AI scrapers.

Why it matters

For webmasters and developers, the takeaway is painfully urgent. Legal terms of service mean nothing to an automated crawler on a deadline. If you do not want your content ingested or your servers overwhelmed, you cannot rely on legalese; you have to lock down your robots.txt file right now.

For the AI industry, this episode highlights the sheer desperation for quality training data. The frontier labs are desperate for high-grade text, and that hunger is spilling over into operational negligence that breaks real-world infrastructure.

We think Anthropic's defense feels remarkably cynical. The company built its entire brand on safety, steerability, and ethical behavior, yet its crawler behaves like an indiscriminate brute-force scraper until someone publicly shames it. Hiding behind a 30-year-old robots.txt standard while ignoring clear legal terms looks less like standard web etiquette and more like convenient plausible deniability. If frontier labs keep hammering independent sites like this, the open web will simply wall itself off completely.