Google, Microsoft, Anthropic and OpenAI Back New AI Crawling Opt-Out Standard

Google, Microsoft, Anthropic and OpenAI announced on October 30, 2026 that they will back a new website crawling standard designed to make it simpler for publishers to stop AI companies from using their content to train models.

Under the current system, a website operator who wants to block AI training scrapers must add a separate entry to their robots.txt file for every individual crawler — a process that grows unwieldy as AI providers run multiple scrapers. The new standard would allow a single robots.txt entry to block all scrapers that collect data for generative AI model training at once. The setting would not apply to models that perform exclusively non-generative tasks.

The four companies have already submitted a draft proposal to the Internet Engineering Task Force, the nonprofit that maintains the Robots Exclusion Protocol — the standardised format that governs how robots.txt files are structured. They plan to incorporate the new crawling standard into that format.

OpenAI and Microsoft intend to add support for the standard in the coming months, while Google plans to follow sometime next year. Meta expressed support for the idea but has not committed to implementing it.

Microsoft also announced a separate feature called AI Grounding, aimed at a different scenario: when an AI model accesses a webpage to answer a user's question rather than to gather training data. Website operators will be able to use the setting to prevent Microsoft's services — and third-party AI tools that rely on Microsoft technologies — from making such requests.