← Back
SiliconANGLE AI

Google, Anthropic and others to support new AI crawling opt-out standard

•
#enterprise
Google, Anthropic and others to support new AI crawling opt-out standard
Level:Intermediate
For:ML Engineers
✦TL;DR

Google, Microsoft, Anthropic, and OpenAI announced they will support an upcoming website crawling opt‑out standard that lets site operators block AI developers from using their content for model training. The approach extends the existing “robots.txt” mechanism to cover AI crawlers. This gives engineers a clear, standardized way to respect content‑ownership preferences when building data pipelines for large‑scale model training.

⚡ Key Takeaways

  • Companies supporting the standard: Google LLC, Microsoft Corp., Anthropic PBC, OpenAI Group PBC
  • Design decision: Extend the traditional “robots.txt” file to signal opt‑out for AI model training crawlers
  • Tradeoff: Enables compliance with content‑owner wishes but may reduce the volume of publicly available data for training
  • Integration: AI developers must configure their crawlers to read and honor the new opt‑out directives in robots.txt
  • Prerequisite: Websites must publish the appropriate opt‑out entries in their robots.txt file for crawlers to detect
💡 Why It Matters

Engineers can now rely on a uniform opt‑out signal to filter training data, simplifying compliance workflows and reducing legal risk when sourcing web content at scale.

✅ Practical Steps

  1. Ensure your web‑crawling pipelines check the robots.txt file for the new AI‑specific opt‑out directives before ingesting content
  2. Update your crawler configuration to skip any URLs flagged by those directives, thereby respecting site owners’ preferences

Want the full story? Read the original article.

Read on SiliconANGLE AI ↗

More like this

MIT affiliates named 2026 Schmidt Sciences AI2050 Fellows

MIT News AI•#enterprise

At Ignite 2026, Microsoft reinvents Copilot as the autonomous engine of enterprise work

SiliconANGLE AI•#llm

Dell puts a single gateway between enterprise AI agents and the data they crave

SiliconANGLE AI•#agents

Your complete guide to AWS Quick at AWS re:Invent 2026

AWS ML Blog•#enterprise

EXPLORE AI NEWS

Daily hand-picked stories on LLMs, RAG, agents and production AI — curated for engineers who ship.

BROWSE NEWS

GET THE WEEKLY DIGEST

Join engineers getting the Monday signal-over-noise AI breakdown. No spam, unsubscribe anytime.

LEARN AI ENGINEERING

Curated courses, research papers, repos and tutorials built for engineers leveling up in AI.

START LEARNING