phenomenon · first seen 8 Oct 2026
web crawling
Also: GPTBot, crawling, AI crawler, web crawler
Automated retrieval of web pages by bots, increasingly driven by AI companies collecting training data.
Mentions per week
PapersCodeDiscussion
Earliest sources
- AI Crawler IndexDiscussion · 8 Oct 2026 · zenincognito · 23 points
- Host-Aware Web Crawling for LLM Pretraining DataPapers · 9 Oct 2026 · Malte Ostendorff
- wojcikdominik/openspiderCode · 13 Oct 2026 · wojcikdominik · 47 stars
- Sarverott/P-jol-rollCode · 13 Oct 2026 · Sarverott · 27 stars
- Show HN: Jevcrawl – A web crawler for AI agents with no LLM in the crawl loopDiscussion · 13 Oct 2026 · rayenmlayeh · 21 points
- mintlify/agent-trafficCode · 10 Nov 2026 · mintlify · 82 stars
Latest sources
- mintlify/agent-trafficCode · 10 Nov 2026 · mintlify · 82 stars
- Show HN: Jevcrawl – A web crawler for AI agents with no LLM in the crawl loopDiscussion · 13 Oct 2026 · rayenmlayeh · 21 points
- Sarverott/P-jol-rollCode · 13 Oct 2026 · Sarverott · 27 stars
- wojcikdominik/openspiderCode · 13 Oct 2026 · wojcikdominik · 47 stars
- Host-Aware Web Crawling for LLM Pretraining DataPapers · 9 Oct 2026 · Malte Ostendorff
- AI Crawler IndexDiscussion · 8 Oct 2026 · zenincognito · 23 points