AI bots are blowing past robots.txt rules
AI companies are quietly bypassing the decades-old robots.txt protocol to scrape publisher sites without permission, according to licensing startup TollBit. The finding follows Forbes recently calling out Perplexity for lifting its investigative work. With web scrapers ignoring traditional courtesy standards, publishers' primary defense against AI crawlers is essentially breaking down.