Google just agreed to adopt the IETF AIPREF standard alongside OpenAI, Anthropic, and Bing. Instead of blocking every AI crawler one by one, website owners will soon be able to use a shared policy token in robots.txt to opt out of training across all participating providers. Google plans to roll out support in early 2027.
Cloudflare just gave AI companies until September 15, 2026, to decouple crawlers used for traditional search from those powering AI training and agents. Miss the deadline, and mixed-use bots will be blocked by default across ad-supported sites. The move puts an end to tech giants scraping free training data under the guise of web indexation, while expanding a model where publishers get paid when content actually creates value.
Google just launched Google-Extended, a tool that lets publishers block their content from training AI models like Bard and Vertex AI while staying visible in Google Search. By adding a simple tag to robots.txt, site owners no longer have to choose between keeping their search traffic or watching their work get scraped for AI models.
