Google joins push for a single AI training switch in robots.txt

If you run a website, your robots.txt file has probably turned into a chaotic mess over the past two years.

Every time a new frontier lab pops up, you have to add another block of crawler rules just to stop them from scraping your work. Miss one, and your data is fair game.

Google announced that it plans to adopt the IETF AIPREF standard, joining OpenAI, Anthropic, and Microsoft's Bing in supporting a unified AI training control.

One token to rule them all

The initiative comes out of the AIPREF working group at the Internet Engineering Task Force. The goal is simple: let website owners express their AI training preferences in a single step instead of managing custom directives for every single provider.

Instead of juggling GPTBot, ClaudeBot, and Google-Extended, admins will soon be able to drop a shared policy token into robots.txt using the user-agent string "__usage_ai-training".

Google has already added a note to its crawler documentation, confirming it plans to officially support this signal in early 2027.

According to Google Search Advocate John Mueller, if a site uses the shared token to opt out and leaves Google-Extended unspecified, Google will treat it as Disallow Google-Extended.

The fine print and the friction

Existing provider-specific directives aren't going anywhere. Website owners can still pick and choose individual permissions.

Google was quick to remind everyone of the catch: sites that explicitly allow Google-Extended remain eligible for grounding in Gemini Apps, meaning those who opt out entirely will miss out on AI-driven referral traffic and clicks.

While the draft aligns with the EU AI Act's GPAI Code of Practice, the standardization process is far from smooth.

Nate Hake, who spent the past year actively participating in the IETF working group, pointed out on X that serious disputes remain. In particular, the current draft largely exempts crawler activity triggered directly by end users, which creators worry leaves a massive loophole for AI spam and copycats.

Why it matters

For site owners and publishers, this is a welcome sanity check, but you shouldn't rush to rewrite your robots.txt this afternoon. Google won't support it until early 2027, and the standard itself is still being debated in working groups.

Tech giants aren't adopting this out of goodwill. They are feeling the heat from European regulators and facing mounting copyright pressure from media publishers worldwide.

We think Google handled this smartly from a corporate standpoint. It scores points for playing nice with open standards while quietly keeping creators hooked on traffic. If you want citations and clicks inside Gemini, you still have to hand your content over through Google-Extended.

The real issue, as always, is enforcement. A single standardized switch makes life easier, but robots.txt remains a gentleman's agreement. It stops the companies willing to comply, but does nothing to deter rogue actors from scraping whatever they want.