Google, Anthropic and others to support new AI crawling opt-out standard
Google, Microsoft, Anthropic, and OpenAI announced they will support an upcoming website crawling opt‑out standard that lets site operators block AI developers from using their content for model training. The approach extends the existing “robots.txt” mechanism to cover AI crawlers. This gives engineers a clear, standardized way to respect content‑ownership preferences when building data pipelines for large‑scale model training.
⚡ Key Takeaways
- Companies supporting the standard: Google LLC, Microsoft Corp., Anthropic PBC, OpenAI Group PBC
- Design decision: Extend the traditional “robots.txt” file to signal opt‑out for AI model training crawlers
- Tradeoff: Enables compliance with content‑owner wishes but may reduce the volume of publicly available data for training
- Integration: AI developers must configure their crawlers to read and honor the new opt‑out directives in robots.txt
- Prerequisite: Websites must publish the appropriate opt‑out entries in their robots.txt file for crawlers to detect
Engineers can now rely on a uniform opt‑out signal to filter training data, simplifying compliance workflows and reducing legal risk when sourcing web content at scale.
✅ Practical Steps
- Ensure your web‑crawling pipelines check the robots.txt file for the new AI‑specific opt‑out directives before ingesting content
- Update your crawler configuration to skip any URLs flagged by those directives, thereby respecting site owners’ preferences
Want the full story? Read the original article.
Read on SiliconANGLE AI ↗