OpenAI launches GPTBot crawler to train future AI

For months, everyone wondered exactly how frontier AI labs were hoovering up the web. Now, OpenAI is giving its scraper a name badge.

OpenAI just rolled out GPTBot, a new web-crawling bot designed to collect publicly available web data to train and improve future AI models, expanding their accuracy and capabilities down the line.

An opt-out switch for site owners

Just like the crawlers operated by Google or Bing to index search pages, GPTBot scours public websites. However, OpenAI outlined clear filtering boundaries for what it collects.

According to OpenAI, the crawler will filter out paywalled content, skip sources known to collect personally identifiable information, and discard text that violates its platform policies.

Critically, webmasters have the power to shut the door. By adding a simple disallow rule to their server's robots.txt file, site owners can tell GPTBot to move along and leave their content alone.

The GPT-5 trademark and legal heat

The launch comes just three weeks after OpenAI filed a trademark application for "GPT-5" with the U.S. Patent and Trademark Office on July 18. That filing covers everything from AI-generated human speech and text to voice recognition and audio-to-text conversion.

Naturally, that got people wondering if GPTBot is directly stocking the pantry for the next frontier model. But you probably shouldn't hold your breath just yet. Back in June, OpenAI CEO Sam Altman said the company is nowhere close to starting training for GPT-5, noting that extensive safety audits need to happen first.

Beyond technical hurdles, OpenAI is also facing mounting global scrutiny over how it gathers training data.

In June, Japan's privacy watchdog issued a warning over unauthorized data harvesting. Italy temporarily banned ChatGPT in April citing EU privacy rules. And in late June, 16 plaintiffs filed a class-action lawsuit alleging OpenAI scraped private data from user interactions, claiming violations under the Computer Fraud and Abuse Act alongside Microsoft.

Why it matters

We see GPTBot as OpenAI's attempt to formalize what used to be a murky scraping operation into an industry-standard protocol.

For creators and website operators, it gives back a sliver of control. You finally have an official target to block in your robots.txt instead of wondering whether your work was fed into an uncredited training run. Still, the opt-out architecture is fundamentally lopsided: OpenAI takes everything by default unless you take proactive action to stop it.

For the broader AI ecosystem, this sets a baseline expectation. As copyright lawsuits and regulatory warnings pile up in places like Japan and Europe, labs will have to be transparent about their crawlers. Crawling cleanly might limit the volume of free data OpenAI can vacuum up, but doing it in the open is likely the only way to survive the coming wave of copyright litigation.