protocol · first seen 8 Nov 2026
robots.txt
Also: robots exclusion protocol
Web standard file where site operators specify which crawlers may access which pages.
Mentions per week
PapersCodeDiscussion
Earliest sources
- ScrapeBench: Evaluating Legal Compliance of AI Agents in Website ScrapingPapers · 8 Nov 2026 · Joseph Marvin Imperial, Daniel Slate, Noam Kolt
Latest sources
- ScrapeBench: Evaluating Legal Compliance of AI Agents in Website ScrapingPapers · 8 Nov 2026 · Joseph Marvin Imperial, Daniel Slate, Noam Kolt