Publishers catch Perplexity scraping behind their backs

Generative AI can take hours of hard shoe-leather reporting and regurgitate it into a polished briefing in about two seconds flat. The publishers who actually broke the news, naturally, are furious.

AI search darling Perplexity finds itself in the crosshairs of multiple major newsrooms. Forbes called out the startup for ripping off an exclusive investigation inside its Perplexity Pages feature. Meanwhile, an investigation by Wired revealed that Perplexity appears to be bypassing the web's standard "do not crawl" protocol to harvest content behind closed doors.

Perplexity, which was raising $250 million at a valuation near $3 billion as of April, insists it did nothing wrong. The startup, backed by Nvidia and Jeff Bezos, claims its summarizations fall squarely within fair use.

Browsing assistant or stealth scraper?

The first battleground is technical: the decades-old Robots Exclusion Protocol (robots.txt), where publishers explicitly tell bots which pages are off-limits.

Wired and independent developer Robb Knight tested Perplexity by feeding it URLs and watching their own server logs. The findings were clear: machines linked to Perplexity visited the URLs anyway, completely ignoring robots.txt rules. On one bare-bones test site Wired created specifically for the probe, the AI even returned the page's contents verbatim.

Perplexity's head of business, Dmitry Shevelenko, offered a clever defense. He told TechCrunch that summarizing a specific link isn't technically "crawling." In his view, crawling means sucking up bulk data to build a general index. When a user pastes a specific URL into the search bar, Perplexity says it is simply acting as an agent visiting that site on the user's behalf.

To publishers, that sounds like pure semantic gymnastics. When an automated service visits prohibited links at the scale of millions of daily prompts, calling it a personal assistant doesn't make it any less of a web scraper.

Nobody owns the facts, but words are copyrighted

The second flashpoint is flat-out plagiarism.

In early June, Forbes broke an exclusive story on former Google CEO Eric Schmidt's secretive military drone startup. The very next day, Perplexity generated an article in its Pages feature that republished the scoop almost entirely. It used nearly identical phrasing, repurposed an illustration made by Forbes' design team, and buried the attribution in tiny, easily missed icons.

In a twist of irony, Wired reported that Perplexity also plagiarized the exact article Wired wrote exposing the startup's scraping practices, lifting one sentence completely verbatim.

Perplexity's defense leans on the argument that "nobody has a monopoly on facts." That is true under U.S. copyright law, which protects original expression rather than underlying information. But as UCLA law professor Mark McKenna pointed out, the moment an AI model replicates the unique phrasing and structural expression of an article, or acts as a direct market substitute for the original piece, the fair use defense starts to crumble.

Why it matters

Perplexity has built one of the slickest productivity tools in AI, but it is running on borrowed time and uncompensated labor.

For users, the product is undeniably tempting. Getting an instant, synthesized answer from multiple paywalled or gated sources saves real time. But the ethical tradeoff is glaring: every time a bot answers your question without sending traffic to the reporter who uncovered the story, the economic engine behind original reporting takes another hit.

Competitively, this draws a sharp contrast with OpenAI. While Sam Altman's crew has been writing multi-million dollar checks to publishers for content licensing, Perplexity has taken a smash-and-grab approach, banking on aggressive legal interpretations of fair use. It is a bold, capital-efficient move that keeps overhead low—right up until the inevitable copyright lawsuits land on its doorstep.

We think Perplexity's "user request" loophole is an ingenious technical argument, but a deeply cynical one. Treating automated scraping as mere user assistance feels like an obvious end-run around long-standing web norms. As courts and publishers push back, the era of scraping the open web without paying the bill is rapidly drawing to a close.