Plagiarism Today · AI Rules · · 2 min read
Four AI giants agree on one way for websites to say no to training
Anthropic, OpenAI, Google and Microsoft have announced a shared setting for a website's robots.txt file. It asks their crawlers not to use the site's content to train AI.
In short: A simpler way to say no, but it is a request, not a lock.
Before this, a website owner who did not want their work used for AI training had to block each company's crawler by name. The new setting replaces that with one rule covering all four companies.
It goes in robots.txt, a small file that tells visiting bots what a website would like them to do. Bots that behave will follow it, but the file cannot physically stop one.
Plagiarism Today points out the limits. The rule covers only these four companies, does not touch content already collected, and does not stop work reaching AI systems through other datasets. The writer sees it as more of a public relations move than a real change.
A robots.txt rule is a polite sign on the door. It is not a lock.
These stories summarise reporting we did not produce. Read the source