Search crawlers versus training crawlers is the clearest pattern in our robots.txt audit: sites block the training bots more often than the bots that feed search answers.
The pairs
Percentages are of the 86 sites whose robots.txt we could read on 29 October 2026, blocking the bot at the root.
| Vendor | Training crawler | Blocked | Search crawler | Blocked |
|---|---|---|---|---|
| OpenAI | GPTBot | 36.0% | OAI-SearchBot | 23.3% |
| Anthropic | ClaudeBot | 46.5% | Claude-SearchBot | 31.4% |
For comparison, Googlebot and bingbot are each blocked by 1.2% of the sites, which is one site.
Reading it
- Publishers seem to separate "do not train on me" from "do not show me in answers", at least in some cases.
- A gap of about 13 points for OpenAI and 15 for Anthropic is large enough to see in a sample this size, and too small to explain without asking the site owners.
- User-triggered fetchers such as ChatGPT-User (26.7%) sit between the two groups.
Limits
Our parser treats the root path only and may differ slightly from each vendor's own rule handling. The 100 sites were chosen by us, not drawn at random. The full table is in the audit write-up and the raw files on the data page.