Google, Microsoft, OpenAI and Anthropic back one-line way to block AI training crawlers
Written by AI from siliconangle.com's report. Not checked by a person.
Google, Microsoft, Anthropic and OpenAI plan to support a crawling standard that lets websites block all AI training scrapers with a single robots.txt entry. OpenAI and Microsoft expect to add support in the coming months, and Google next year.
Google, Microsoft, Anthropic and OpenAI announced plans to support an upcoming crawling standard that lets website operators stop AI developers from using their content to train models. Under the current system, a company that wants to disallow training-related scraping must edit its robots.txt file many times, adding a separate entry for every AI provider and a further entry for each scraper when a provider runs several. The new standard would let operators block all scrapers that collect training data for generative AI models with one robots.txt entry. The setting would not apply to models that perform exclusively non-generative tasks.
The four companies plan to build the standard into the Robots Exclusion Protocol, the format that robots.txt files follow. They have already submitted a draft of their proposal to the Internet Engineering Task Force, the nonprofit that maintains the protocol.
Microsoft said in a blog post that it will also offer a setting called AI Grounding. It covers cases where an AI model visits a webpage to answer a user's question rather than to collect training data. Website operators will be able to stop Microsoft's services from sending such requests, and the control will also extend to third-party AI tools that use Microsoft technologies to process prompts. OpenAI plans to offer similar controls.
OpenAI and Microsoft intend to support the standard in the coming months, while Google will follow sometime next year. Meta Platforms said it supports the idea behind the technology but has not yet committed to implementing it.
