Cloudflare catches Perplexity bypassing website scrape blocks

What happens when you tell an AI crawler to stay off your lawn, and it puts on a fake mustache to sneak in anyway?

That is essentially what internet infrastructure giant Cloudflare accused Perplexity of doing on Monday. In a newly published report, Cloudflare said it caught the AI search darling actively disguising its identity to scrape websites that had explicitly forbidden it.

Impersonating Chrome to bypass blocks

According to Cloudflare, the investigation began after multiple customers complained that Perplexity was still scraping their sites, even after they added strict blocks in their Robots.txt files targeting Perplexity’s known crawlers.

When Cloudflare set up tests to investigate, they spotted a deliberate evasive maneuver. When Perplexity’s declared bots hit a wall, the company allegedly rotated its user-agent strings to impersonate a generic Google Chrome browser running on macOS. It also shuffled its Autonomous System Networks (ASNs) to hide the network origin of the requests.

Cloudflare stated that it used machine learning and behavioral network fingerprinting to track the activity, finding that Perplexity was firing millions of unauthorized requests per day across tens of thousands of domains.

Perplexity fired back immediately. Spokesperson Jesse Dwyer dismissed Cloudflare’s findings as a marketing pitch, telling TechCrunch that the post’s own screenshots showed that no content was actually accessed. In a follow-up email, Dwyer went even further, claiming the bot identified by Cloudflare was not even theirs.

Cloudflare has since stripped Perplexity’s crawlers from its verified list and deployed new techniques to block them.

The crumbling handshake of the open web

This is not unfamiliar territory for Perplexity. Last year, outlets like Wired accused the startup of plagiarizing publisher content. Weeks later at the Disrupt 2024 conference, Perplexity CEO Aravind Srinivas awkwardly hesitated when asked on stage to define plagiarism.

Meanwhile, Cloudflare has been positioning itself as the toll collector of the AI era. Just last month, Cloudflare rolled out a marketplace enabling publishers and site owners to charge AI companies for scraping their content. Cloudflare CEO Matthew Prince warned that AI scraping is breaking the economic foundation of the web, following a free anti-scraping tool the company launched last year.

Why it matters

For anyone relying on Perplexity for snappy research, this exposes the fragile supply chain powering modern answer engines. The quick summaries users enjoy rely on round-the-clock scraping, and that pipeline is drying up as publishers harden their defenses.

In our view, Perplexity’s alleged tactics cross a dangerous line. Disguising commercial crawlers as ordinary desktop browsers ignores decades of web etiquette. Robots.txt was always an honor system, but openly flouting it destroys whatever goodwill AI companies had left with content creators.

It also hands an enormous win to Cloudflare. By calling out Perplexity right after launching its pay-to-scrape marketplace, Cloudflare looks like the indispensable protector of publishers. The message to AI startups is clear: the era of free, quiet scraping is over, and your options are either to pay up at the front door or get locked out entirely.