Google gives publishers an AI training opt-out without killing search rank

If you run a website, you have probably spent the past few months in an impossible bind: tech giants are scraping your articles to train their AI models, but blocking Google entirely means disappearing from search.

Now, Google is finally separating the two.

Google just introduced Google-Extended, a new control that allows website publishers to opt out of having their content used to train the company's AI models while keeping their pages indexed in Google Search.

Search yes, training no

Publishers can activate the toggle by adding Google-Extended to their robots.txt file, the standard text file that tells web crawlers what they can and cannot touch.

Google confirmed back in July that it trains its Bard chatbot on publicly scraped web data. Until now, publishers had no native technical tool to separate Google's search crawler from its generative AI appetite.

According to Google, the new flag lets sites manage whether their content helps improve Bard and the generative APIs powering Vertex AI.

Google also noted that as AI tools continue to expand, it plans to explore additional machine-readable controls for publishers and will share more soon.

The standoff with web publishers

Content creators and media heavyweights have been revolting against AI scrapers for months.

Outlets like The New York Times, CNN, Reuters, and Medium already blocked OpenAI's web crawlers to keep their archives out of ChatGPT. But Google presented a much tougher problem, because shutting out Googlebot meant sacrificing critical search traffic.

Without a dedicated toggle, publishers had to resort to legal maneuvers. The New York Times, for instance, updated its terms of service to outright ban companies from training AI on its reporting.

Google-Extended provides the technical compromise the industry has been demanding.

Why it matters

For publishers and independent creators, this removes a toxic dilemma. You no longer have to sacrifice your primary source of web traffic just to protect your intellectual property from becoming model weights. If you care about your content footprint, setting up Google-Extended in your robots.txt is a no-brainer step to take right now.

For the AI landscape, Google is essentially buying goodwill and attempting to head off legal threats. With OpenAI facing widespread pushback, Google looks smart by giving publishers an easy opt-out mechanism before regulators or lawsuits force its hand.

However, we think this is only a partial fix. A robots.txt directive relies entirely on the honor system, with zero independent auditing to prove the data was actually excluded. More importantly, it does nothing about the years of archives Google already used to train its current generation of models. It protects tomorrow's content, but yesterday's work is already baked in.