The New York Times blocks OpenAI's GPTBot
The New York Times just blocked OpenAI’s GPTBot web crawler via its robots.txt file, cutting off access to its reporting for training AI models. Archive records indicate the block took effect around August 17th. Why it matters: Media publishers are drawing hard lines around their content, with reports indicating the Times is already considering a copyright lawsuit against OpenAI.
Media giants are drawing a line in the sand against AI scrapers.
The Times shuts out GPTBot
The New York Times has officially blocked OpenAI's web crawler, GPTBot. A quick look at the publication's robots.txt page shows that the bot, which OpenAI introduced earlier this month, is now explicitly disallowed.
According to the Internet Archive's Wayback Machine, the block appears to have gone live as early as August 17th.
The technical block follows a policy update from earlier this month, where the Times amended its terms of service to prohibit using its content to train AI models.
Things might also escalate to the courtroom. NPR reported last week that the Times is weighing legal action against OpenAI over alleged intellectual property violations. If it files suit, it will join figures like author Sarah Silverman, who sued in July over the Books3 dataset, and lawyer Matthew Butterick, who contends OpenAI's scraping amounts to piracy.
Why it matters
The era of freely scraping high-quality journalism for AI training looks to be crumbling.
By slamming the door shut with robots.txt and threatening lawsuits, the Times is sending a clear warning to AI labs. In our view, high-profile publishers will increasingly demand licensing revenue rather than tolerating unchecked data harvesting.
For OpenAI and its rivals, the pipeline of free, authoritative language data is shrinking fast. The choice moving forward is becoming stark: pay up for content deals, or face a mounting wall of legal headaches.
Sources (1)
- theverge.com(opens in a new tab)Originaltheverge.com