We checked the robots.txt of 100 well-known sites on 29 October 2026 to see which of them block AI crawlers, and the short answer is that blocking training bots is common while blocking Googlebot or bingbot is almost unheard of.
The result in one table
86 of the 100 sites returned a readable robots.txt. The table shows how many of those 86 disallow the bot from the site root.
| Bot | Sites blocking | Share of 86 |
|---|---|---|
| CCBot (Common Crawl) | 43 | 50.0% |
| ClaudeBot | 40 | 46.5% |
| Bytespider | 40 | 46.5% |
| Applebot-Extended | 37 | 43.0% |
| Meta-ExternalAgent | 36 | 41.9% |
| PerplexityBot | 34 | 39.5% |
| GPTBot | 31 | 36.0% |
| Google-Extended | 29 | 33.7% |
| Claude-SearchBot | 27 | 31.4% |
| Perplexity-User | 25 | 29.1% |
| ChatGPT-User | 23 | 26.7% |
| MistralAI-User | 22 | 25.6% |
| OAI-SearchBot | 20 | 23.3% |
| YandexBot | 10 | 11.6% |
| Kimi-User | 9 | 10.5% |
| KimiBot, Kimi-SearchBot | 8 | 9.3% |
| Googlebot | 1 | 1.2% |
| bingbot | 1 | 1.2% |
Only 15 of the 86 sites (17.4%) served an llms.txt file.
What stands out
- Training crawlers are blocked more than search crawlers. GPTBot (training) is blocked on 31 sites, OAI-SearchBot (search) on 20. For Claude the same pattern holds: ClaudeBot 40, Claude-SearchBot 27.
- Few publishers cut themselves off from classic search. Googlebot and bingbot are each blocked on one site.
- The user-triggered bots are blocked less often than the training bots, but still on roughly a quarter of sites.
- llms.txt is rare. It is present on fewer than one site in five in this sample.
Method
We used a short Python script. For each domain it downloaded https://<domain>/robots.txt and https://<domain>/llms.txt, kept the robots file only if it was plain text with HTTP 200, and asked Python's urllib.robotparser whether each bot may fetch the root path /. The sites are 100 domains chosen across news, e-commerce, technology, SEO, reference and public institutions, in English and French. The script and the full list are on the data page.
The 14 sites we could not analyse
cnn.com, stackoverflow.com, walmart.com, fnac.com, lequipe.fr, darty.com, cultura.com, bestbuy.com, adidas.com, npr.org, pcmag.com, sciencedirect.com, nih.gov and europa.eu did not return a readable robots.txt to our request (many protect against automated clients). They are counted apart and excluded from every percentage.
Limits
- Python's robots.txt parser can differ slightly from Google's own handling of Allow and Disallow rules.
- We only test the root path. A site may block a bot on subfolders and show as "allowed" here.
- robots.txt states a policy. It does not prove that a bot obeys it, and it says nothing about firewall-level blocking.
- 100 sites chosen by us is not a random sample of the web.
- This is a single snapshot taken on 29 October 2026. We plan to repeat it and will publish changes on the updates page.
Download
The raw per-bot results are in resultats.csv, the totals in stats.csv, and both are linked from the data page.