Reddit blocks the Wayback Machine to stop AI scraping

If you wanted to see what internet dwellers were arguing about on Reddit a decade ago, the Wayback Machine was always the place to go. Not anymore.

Reddit is severely restricting the Internet Archive from crawling its pages. The reason? Reddit says it caught AI companies using the non-profit's historical snapshots as a backdoor to scrape its vast troves of human conversation for model training.

The front page is all you get

Starting today, Reddit is clamping down hard on what the Wayback Machine can see.

The Internet Archive will no longer be allowed to index post detail pages, comment threads, or user profiles. The only thing left untouched is Reddit's homepage.

In practice, that means future internet historians will only be able to see which headlines were trending on Reddit on any given day, while the actual discussions and community reactions are wiped from the public record.

Reddit spokesperson Tim Rathschmidt told The Verge that until the Archive can defend its site and comply with platform policies—like respecting user privacy and purging deleted content—access has to be limited to protect redditors. Wayback Machine director Mark Graham stated that both parties have a longstanding relationship and are actively discussing the issue.

Pay up or get locked out

Let’s be real: Reddit's war on scrapers has never really been about privacy. It is about cold, hard cash.

Over the past two years, Reddit has systematically locked down its data. It hiked API prices in 2023, blocked major search engines that refused to pay, and inked multi-million-dollar training deals with Google and OpenAI. Just this past June, it sued Anthropic for allegedly scraping Reddit data after claiming it stopped.

To Reddit, its endless archives of conversational data are the crown jewels of generative AI. If Big Tech giants have to pay tens of millions to touch it, Reddit simply cannot tolerate a free, non-profit digital library acting as an accidental data pipeline for AI competitors.

Why it matters

We think this move shows just how thoroughly the AI gold rush is collateralizing the open web.

For everyday users, researchers, and journalists, this is an undeniable blow. The Internet Archive was built to preserve human culture and keep platforms accountable when posts mysteriously vanish. By cutting off comment sections and threads, Reddit is effectively locking its cultural history behind a private paywall.

From a business perspective, Reddit's strategy is undeniably shrewd. By plugging this loophole, it shuts down another backdoor for AI startups trying to train models on the cheap, forcing everyone toward its paid licensing table.

Yet the long-term cost looks steep. Framing the blockade as "user protection" while selling that exact same user content to OpenAI and Google feels deeply hypocritical. As platforms continue to cordon off their data from public memory, the shared history of the internet is turning into proprietary real estate.