Uptime Report Research · Robots.txt census, November 2026

16.7% of the Top 10,000 Websites Block GPTBot by Name, but Only 4.1% Block ChatGPT Search

Do the busiest websites block GPTBot? Most do not: as of 11 November 2026, 1,407 of the 8,404 top Chrome origins we could read (16.7%) block GPTBot by name in robots.txt, while only 345 (4.1%) block OAI-SearchBot, the agent that decides inclusion in ChatGPT search. We fetched robots.txt for all 10,000 origins in Google’s CrUX top-10,000 list and parsed 25 user agents.

Key findings

  1. 16.7% of the 8,404 top origins we could read block GPTBot by name in robots.txt; in the top 1,000 the rate is 20.3%.
  2. Only 4.1% block OAI-SearchBot by name. Of the sites that block GPTBot, 25.9% also block ChatGPT search.
  3. 21.1% fully block at least one AI training agent, and 58.6% of those leave every AI search agent we tracked allowed.
  4. The most blocked AI tokens by name are GPTBot (16.7%), CCBot (16.6%) and Bytespider (15.8%).
  5. News publishers are the outlier: 34.8% block GPTBot by name, against 1.9% of government sites.
  6. 8.5% of checked top sites serve a valid /llms.txt file (671 sites); among sites that name AI agents without fully blocking any, the rate is 39.0%.
  7. 327 sites (3.9%) close robots.txt to every crawler, Googlebot included, so AI agents are blocked there without being named.
16.7%block GPTBot by name
4.1%block OAI-SearchBot by name
20.3%of the top 1,000 block GPTBot
8.5%serve a valid /llms.txt
8,404origins measured of 10,000 fetched
25user agents classified per site

Agent by agent: who gets blocked, by rank and site type

Each cell is the share of sites whose robots.txt names that agent in its own group and disallows the whole site. Training crawlers sit at the top; the gap to the search and user-fetch rows is the main pattern. GPTBot is the most blocked token by name at 16.7%, followed by CCBot (16.6%) and Bytespider (15.8%); OpenAI’s search agent is blocked by 4.1%.

Alln=8,404Top 1kn=8631k to 5kn=3,3725k to 10kn=4,169Newsn=768Retailn=604Govn=312Edun=76Othern=6,644TRAININGSEARCHUSEROTHERGPTBotGPTBot, All measured: 16.7% block by name (8,404 sites)17GPTBot, Top 1,000: 20.3% block by name (863 sites)20GPTBot, 1,001 to 5,000: 17.5% block by name (3,372 sites)18GPTBot, 5,001 to 10,000: 15.4% block by name (4,169 sites)15GPTBot, News publishers: 34.8% block by name (768 sites)35GPTBot, Retail and e-commerce: 17.9% block by name (604 sites)18GPTBot, Government: 1.9% block by name (312 sites)2GPTBot, Education: 3.9% block by name (76 sites)4GPTBot, Everything else: 15.4% block by name (6,644 sites)15ClaudeBotClaudeBot, All measured: 15.2% block by name (8,404 sites)15ClaudeBot, Top 1,000: 18.9% block by name (863 sites)19ClaudeBot, 1,001 to 5,000: 15.9% block by name (3,372 sites)16ClaudeBot, 5,001 to 10,000: 13.8% block by name (4,169 sites)14ClaudeBot, News publishers: 32.0% block by name (768 sites)32ClaudeBot, Retail and e-commerce: 15.9% block by name (604 sites)16ClaudeBot, Government: 1.9% block by name (312 sites)2ClaudeBot, Education: 2.6% block by name (76 sites)3ClaudeBot, Everything else: 13.9% block by name (6,644 sites)14Google-ExtendedGoogle-Extended, All measured: 13.5% block by name (8,404 sites)13Google-Extended, Top 1,000: 17.1% block by name (863 sites)17Google-Extended, 1,001 to 5,000: 14.0% block by name (3,372 sites)14Google-Extended, 5,001 to 10,000: 12.3% block by name (4,169 sites)12Google-Extended, News publishers: 26.7% block by name (768 sites)27Google-Extended, Retail and e-commerce: 13.2% block by name (604 sites)13Google-Extended, Government: 2.2% block by name (312 sites)2Google-Extended, Education: 3.9% block by name (76 sites)4Google-Extended, Everything else: 12.6% block by name (6,644 sites)13Applebot-ExtendedApplebot-Extended, All measured: 12.5% block by name (8,404 sites)13Applebot-Extended, Top 1,000: 14.6% block by name (863 sites)15Applebot-Extended, 1,001 to 5,000: 12.8% block by name (3,372 sites)13Applebot-Extended, 5,001 to 10,000: 11.9% block by name (4,169 sites)12Applebot-Extended, News publishers: 25.4% block by name (768 sites)25Applebot-Extended, Retail and e-commerce: 6.5% block by name (604 sites)6Applebot-Extended, Government: 2.2% block by name (312 sites)2Applebot-Extended, Education: 3.9% block by name (76 sites)4Applebot-Extended, Everything else: 12.2% block by name (6,644 sites)12Meta-ExternalAgentMeta-ExternalAgent, All measured: 12.8% block by name (8,404 sites)13Meta-ExternalAgent, Top 1,000: 15.4% block by name (863 sites)15Meta-ExternalAgent, 1,001 to 5,000: 13.2% block by name (3,372 sites)13Meta-ExternalAgent, 5,001 to 10,000: 12.0% block by name (4,169 sites)12Meta-ExternalAgent, News publishers: 26.4% block by name (768 sites)26Meta-ExternalAgent, Retail and e-commerce: 9.8% block by name (604 sites)10Meta-ExternalAgent, Government: 1.6% block by name (312 sites)2Meta-ExternalAgent, Education: 2.6% block by name (76 sites)3Meta-ExternalAgent, Everything else: 12.2% block by name (6,644 sites)12AmazonbotAmazonbot, All measured: 13.4% block by name (8,404 sites)13Amazonbot, Top 1,000: 16.9% block by name (863 sites)17Amazonbot, 1,001 to 5,000: 13.6% block by name (3,372 sites)14Amazonbot, 5,001 to 10,000: 12.5% block by name (4,169 sites)12Amazonbot, News publishers: 27.6% block by name (768 sites)28Amazonbot, Retail and e-commerce: 11.9% block by name (604 sites)12Amazonbot, Government: 2.9% block by name (312 sites)3Amazonbot, Education: 2.6% block by name (76 sites)3Amazonbot, Everything else: 12.5% block by name (6,644 sites)12CCBotCCBot, All measured: 16.6% block by name (8,404 sites)17CCBot, Top 1,000: 21.1% block by name (863 sites)21CCBot, 1,001 to 5,000: 17.3% block by name (3,372 sites)17CCBot, 5,001 to 10,000: 15.2% block by name (4,169 sites)15CCBot, News publishers: 37.4% block by name (768 sites)37CCBot, Retail and e-commerce: 15.9% block by name (604 sites)16CCBot, Government: 2.2% block by name (312 sites)2CCBot, Education: 2.6% block by name (76 sites)3CCBot, Everything else: 15.2% block by name (6,644 sites)15OAI-SearchBotOAI-SearchBot, All measured: 4.1% block by name (8,404 sites)4OAI-SearchBot, Top 1,000: 6.7% block by name (863 sites)7OAI-SearchBot, 1,001 to 5,000: 4.2% block by name (3,372 sites)4OAI-SearchBot, 5,001 to 10,000: 3.5% block by name (4,169 sites)3OAI-SearchBot, News publishers: 18.9% block by name (768 sites)19OAI-SearchBot, Retail and e-commerce: 4.5% block by name (604 sites)4OAI-SearchBot, Government: 0.0% block by name (312 sites)0OAI-SearchBot, Education: 0.0% block by name (76 sites)0OAI-SearchBot, Everything else: 2.6% block by name (6,644 sites)3Claude-SearchBotClaude-SearchBot, All measured: 3.2% block by name (8,404 sites)3Claude-SearchBot, Top 1,000: 7.2% block by name (863 sites)7Claude-SearchBot, 1,001 to 5,000: 3.1% block by name (3,372 sites)3Claude-SearchBot, 5,001 to 10,000: 2.5% block by name (4,169 sites)3Claude-SearchBot, News publishers: 13.9% block by name (768 sites)14Claude-SearchBot, Retail and e-commerce: 4.6% block by name (604 sites)5Claude-SearchBot, Government: 0.0% block by name (312 sites)0Claude-SearchBot, Education: 0.0% block by name (76 sites)0Claude-SearchBot, Everything else: 2.0% block by name (6,644 sites)2Meta-WebIndexerMeta-WebIndexer, All measured: 1.8% block by name (8,404 sites)2Meta-WebIndexer, Top 1,000: 3.5% block by name (863 sites)3Meta-WebIndexer, 1,001 to 5,000: 1.7% block by name (3,372 sites)2Meta-WebIndexer, 5,001 to 10,000: 1.5% block by name (4,169 sites)2Meta-WebIndexer, News publishers: 7.0% block by name (768 sites)7Meta-WebIndexer, Retail and e-commerce: 4.6% block by name (604 sites)5Meta-WebIndexer, Government: 0.0% block by name (312 sites)0Meta-WebIndexer, Education: 0.0% block by name (76 sites)0Meta-WebIndexer, Everything else: 1.1% block by name (6,644 sites)1Amzn-SearchBotAmzn-SearchBot, All measured: 1.2% block by name (8,404 sites)1Amzn-SearchBot, Top 1,000: 2.3% block by name (863 sites)2Amzn-SearchBot, 1,001 to 5,000: 1.0% block by name (3,372 sites)1Amzn-SearchBot, 5,001 to 10,000: 1.2% block by name (4,169 sites)1Amzn-SearchBot, News publishers: 5.9% block by name (768 sites)6Amzn-SearchBot, Retail and e-commerce: 0.8% block by name (604 sites)1Amzn-SearchBot, Government: 0.0% block by name (312 sites)0Amzn-SearchBot, Education: 0.0% block by name (76 sites)0Amzn-SearchBot, Everything else: 0.8% block by name (6,644 sites)1PerplexityBotPerplexityBot, All measured: 6.8% block by name (8,404 sites)7PerplexityBot, Top 1,000: 12.9% block by name (863 sites)13PerplexityBot, 1,001 to 5,000: 7.0% block by name (3,372 sites)7PerplexityBot, 5,001 to 10,000: 5.4% block by name (4,169 sites)5PerplexityBot, News publishers: 25.5% block by name (768 sites)26PerplexityBot, Retail and e-commerce: 8.1% block by name (604 sites)8PerplexityBot, Government: 0.6% block by name (312 sites)1PerplexityBot, Education: 0.0% block by name (76 sites)0PerplexityBot, Everything else: 4.9% block by name (6,644 sites)5DuckAssistBotDuckAssistBot, All measured: 2.8% block by name (8,404 sites)3DuckAssistBot, Top 1,000: 6.5% block by name (863 sites)6DuckAssistBot, 1,001 to 5,000: 3.0% block by name (3,372 sites)3DuckAssistBot, 5,001 to 10,000: 1.9% block by name (4,169 sites)2DuckAssistBot, News publishers: 12.8% block by name (768 sites)13DuckAssistBot, Retail and e-commerce: 7.1% block by name (604 sites)7DuckAssistBot, Government: 0.0% block by name (312 sites)0DuckAssistBot, Education: 0.0% block by name (76 sites)0DuckAssistBot, Everything else: 1.5% block by name (6,644 sites)1ChatGPT-UserChatGPT-User, All measured: 6.1% block by name (8,404 sites)6ChatGPT-User, Top 1,000: 8.8% block by name (863 sites)9ChatGPT-User, 1,001 to 5,000: 6.7% block by name (3,372 sites)7ChatGPT-User, 5,001 to 10,000: 5.1% block by name (4,169 sites)5ChatGPT-User, News publishers: 21.6% block by name (768 sites)22ChatGPT-User, Retail and e-commerce: 7.5% block by name (604 sites)7ChatGPT-User, Government: 0.6% block by name (312 sites)1ChatGPT-User, Education: 0.0% block by name (76 sites)0ChatGPT-User, Everything else: 4.6% block by name (6,644 sites)5Claude-UserClaude-User, All measured: 3.5% block by name (8,404 sites)3Claude-User, Top 1,000: 7.2% block by name (863 sites)7Claude-User, 1,001 to 5,000: 3.5% block by name (3,372 sites)3Claude-User, 5,001 to 10,000: 2.6% block by name (4,169 sites)3Claude-User, News publishers: 14.2% block by name (768 sites)14Claude-User, Retail and e-commerce: 6.5% block by name (604 sites)6Claude-User, Government: 0.0% block by name (312 sites)0Claude-User, Education: 0.0% block by name (76 sites)0Claude-User, Everything else: 2.1% block by name (6,644 sites)2Meta-ExternalFetcherMeta-ExternalFetcher, All measured: 3.4% block by name (8,404 sites)3Meta-ExternalFetcher, Top 1,000: 5.4% block by name (863 sites)5Meta-ExternalFetcher, 1,001 to 5,000: 3.6% block by name (3,372 sites)4Meta-ExternalFetcher, 5,001 to 10,000: 2.7% block by name (4,169 sites)3Meta-ExternalFetcher, News publishers: 14.2% block by name (768 sites)14Meta-ExternalFetcher, Retail and e-commerce: 4.3% block by name (604 sites)4Meta-ExternalFetcher, Government: 0.0% block by name (312 sites)0Meta-ExternalFetcher, Education: 0.0% block by name (76 sites)0Meta-ExternalFetcher, Everything else: 2.2% block by name (6,644 sites)2Amzn-UserAmzn-User, All measured: 1.5% block by name (8,404 sites)2Amzn-User, Top 1,000: 2.2% block by name (863 sites)2Amzn-User, 1,001 to 5,000: 1.6% block by name (3,372 sites)2Amzn-User, 5,001 to 10,000: 1.3% block by name (4,169 sites)1Amzn-User, News publishers: 5.7% block by name (768 sites)6Amzn-User, Retail and e-commerce: 0.3% block by name (604 sites)0Amzn-User, Government: 0.0% block by name (312 sites)0Amzn-User, Education: 0.0% block by name (76 sites)0Amzn-User, Everything else: 1.2% block by name (6,644 sites)1Perplexity-UserPerplexity-User, All measured: 3.5% block by name (8,404 sites)4Perplexity-User, Top 1,000: 7.2% block by name (863 sites)7Perplexity-User, 1,001 to 5,000: 3.6% block by name (3,372 sites)4Perplexity-User, 5,001 to 10,000: 2.7% block by name (4,169 sites)3Perplexity-User, News publishers: 14.6% block by name (768 sites)15Perplexity-User, Retail and e-commerce: 6.6% block by name (604 sites)7Perplexity-User, Government: 0.0% block by name (312 sites)0Perplexity-User, Education: 0.0% block by name (76 sites)0Perplexity-User, Everything else: 2.2% block by name (6,644 sites)2MistralAI-UserMistralAI-User, All measured: 2.5% block by name (8,404 sites)2MistralAI-User, Top 1,000: 5.4% block by name (863 sites)5MistralAI-User, 1,001 to 5,000: 2.4% block by name (3,372 sites)2MistralAI-User, 5,001 to 10,000: 1.9% block by name (4,169 sites)2MistralAI-User, News publishers: 10.2% block by name (768 sites)10MistralAI-User, Retail and e-commerce: 4.5% block by name (604 sites)4MistralAI-User, Government: 0.0% block by name (312 sites)0MistralAI-User, Education: 0.0% block by name (76 sites)0MistralAI-User, Everything else: 1.6% block by name (6,644 sites)2anthropic-aianthropic-ai, All measured: 13.0% block by name (8,404 sites)13anthropic-ai, Top 1,000: 15.9% block by name (863 sites)16anthropic-ai, 1,001 to 5,000: 13.3% block by name (3,372 sites)13anthropic-ai, 5,001 to 10,000: 12.1% block by name (4,169 sites)12anthropic-ai, News publishers: 29.7% block by name (768 sites)30anthropic-ai, Retail and e-commerce: 3.5% block by name (604 sites)3anthropic-ai, Government: 1.6% block by name (312 sites)2anthropic-ai, Education: 2.6% block by name (76 sites)3anthropic-ai, Everything else: 12.6% block by name (6,644 sites)13Claude-WebClaude-Web, All measured: 11.4% block by name (8,404 sites)11Claude-Web, Top 1,000: 13.8% block by name (863 sites)14Claude-Web, 1,001 to 5,000: 12.0% block by name (3,372 sites)12Claude-Web, 5,001 to 10,000: 10.4% block by name (4,169 sites)10Claude-Web, News publishers: 24.3% block by name (768 sites)24Claude-Web, Retail and e-commerce: 4.5% block by name (604 sites)4Claude-Web, Government: 1.6% block by name (312 sites)2Claude-Web, Education: 2.6% block by name (76 sites)3Claude-Web, Everything else: 11.0% block by name (6,644 sites)11BytespiderBytespider, All measured: 15.8% block by name (8,404 sites)16Bytespider, Top 1,000: 20.5% block by name (863 sites)21Bytespider, 1,001 to 5,000: 16.4% block by name (3,372 sites)16Bytespider, 5,001 to 10,000: 14.4% block by name (4,169 sites)14Bytespider, News publishers: 34.2% block by name (768 sites)34Bytespider, Retail and e-commerce: 16.6% block by name (604 sites)17Bytespider, Government: 2.2% block by name (312 sites)2Bytespider, Education: 2.6% block by name (76 sites)3Bytespider, Everything else: 14.4% block by name (6,644 sites)14

Percent of sites in each column that fully block the agent by name (wildcard-only blocks excluded). Darker is higher. Hover a cell for the exact value and sample size. Categories follow the rule in the methodology. Columns with fewer than 60 sites are indicative only.

Rank matters. In the top 1,000 band, 20.3% block GPTBot by name, against 15.4% of origins ranked 5,001 to 10,000. Category matters more: 34.8% of news publishers block GPTBot by name and 18.9% block OAI-SearchBot, while government sites sit at 1.9%.

Training versus search: the split most sites make

OpenAI, Anthropic, Meta and Amazon each document separate user agents for training, for AI search indexing and for fetches a user asks for; Google and Apple use a training control token next to their search crawler. The dot plot compares how often sites block each by name. Within a vendor, the training dot (red) sits far to the right of the search dot (blue) for OpenAI, Anthropic, Meta and Amazon.

0%5%10%15%OpenAIGPTBot (training): 16.7% block by nameOAI-SearchBot (search): 4.1% block by nameChatGPT-User (user fetch): 6.1% block by nameGPTBot 16.7%AnthropicClaudeBot (training): 15.2% block by nameClaude-SearchBot (search): 3.2% block by nameClaude-User (user fetch): 3.5% block by nameClaudeBot 15.2%MetaMeta-ExternalAgent (training): 12.8% block by nameMeta-WebIndexer (search): 1.8% block by nameMeta-ExternalFetcher (user fetch): 3.4% block by nameMeta-ExternalAgent 12.8%AmazonAmazonbot (training): 13.4% block by nameAmzn-SearchBot (search): 1.2% block by nameAmzn-User (user fetch): 1.5% block by nameAmazonbot 13.4%PerplexityPerplexityBot (search): 6.8% block by namePerplexity-User (user fetch): 3.5% block by namePerplexityBot 6.8%GoogleGoogle-Extended (training): 13.5% block by nameGooglebot (search, classic search crawler): 0.3% block by nameGoogle-Extended 13.5%AppleApplebot-Extended (training): 12.5% block by nameApplebot (search, classic search crawler): 1.4% block by nameApplebot-Extended 12.5%

training agentsearch agent (Googlebot and Applebot for Google and Apple)user-initiated fetch

Of the 1,407 sites that block GPTBot by name, 365 (25.9%) also block OAI-SearchBot and 537 (38.2%) also block ChatGPT-User. Across vendors, 1,774 sites (21.1%) fully block at least one training agent; 1,039 of them (58.6%) leave all six AI search agents in our list allowed. News publishers in the top 5,000 that block GPTBot but leave OAI-SearchBot allowed include abcnews.com, apnews.com, collider.com, crex.com, eiga.com, gamerant.com, gendai.media, iol.co.za, movieweb.com, news.ltn.com.tw. News publishers in the same range that block both include actu.fr, amp.elperiodico.com, amp.marca.com, amp.sport.es, cnnespanol.cnn.com, edition.cnn.com.

Policy mix by rank band and category

Top 1,000Top 1,000: Blocks all crawlers, Googlebot included, 44 of 863 (5.1%)Top 1,000: Blocks AI training and AI search, 119 of 863 (13.8%)14%Top 1,000: Blocks AI training, allows AI search, 109 of 863 (12.6%)13%Top 1,000: Blocks only AI search or user agents, 9 of 863 (1.0%)Top 1,000: Names AI agents, no full block, 40 of 863 (4.6%)Top 1,000: No AI-specific rules, 542 of 863 (62.8%)63%1,001 to 5,0001,001 to 5,000: Blocks all crawlers, Googlebot included, 120 of 3,372 (3.6%)1,001 to 5,000: Blocks AI training and AI search, 305 of 3,372 (9.0%)9%1,001 to 5,000: Blocks AI training, allows AI search, 438 of 3,372 (13.0%)13%1,001 to 5,000: Blocks only AI search or user agents, 28 of 3,372 (0.8%)1,001 to 5,000: Names AI agents, no full block, 151 of 3,372 (4.5%)1,001 to 5,000: No AI-specific rules, 2,330 of 3,372 (69.1%)69%5,001 to 10,0005,001 to 10,000: Blocks all crawlers, Googlebot included, 163 of 4,169 (3.9%)5,001 to 10,000: Blocks AI training and AI search, 311 of 4,169 (7.5%)7%5,001 to 10,000: Blocks AI training, allows AI search, 492 of 4,169 (11.8%)12%5,001 to 10,000: Blocks only AI search or user agents, 14 of 4,169 (0.3%)5,001 to 10,000: Names AI agents, no full block, 177 of 4,169 (4.2%)5,001 to 10,000: No AI-specific rules, 3,012 of 4,169 (72.2%)72%News publishersNews publishers: Blocks AI training and AI search, 243 of 768 (31.6%)32%News publishers: Blocks AI training, allows AI search, 123 of 768 (16.0%)16%News publishers: Blocks only AI search or user agents, 1 of 768 (0.1%)News publishers: Names AI agents, no full block, 50 of 768 (6.5%)7%News publishers: No AI-specific rules, 351 of 768 (45.7%)46%Retail and e-commerceRetail and e-commerce: Blocks AI training and AI search, 60 of 604 (9.9%)10%Retail and e-commerce: Blocks AI training, allows AI search, 76 of 604 (12.6%)13%Retail and e-commerce: Blocks only AI search or user agents, 18 of 604 (3.0%)Retail and e-commerce: Names AI agents, no full block, 83 of 604 (13.7%)14%Retail and e-commerce: No AI-specific rules, 367 of 604 (60.8%)61%GovernmentGovernment: Blocks all crawlers, Googlebot included, 20 of 312 (6.4%)6%Government: Blocks AI training and AI search, 3 of 312 (1.0%)Government: Blocks AI training, allows AI search, 7 of 312 (2.2%)Government: Names AI agents, no full block, 2 of 312 (0.6%)Government: No AI-specific rules, 280 of 312 (89.7%)90%EducationEducation: Blocks all crawlers, Googlebot included, 2 of 76 (2.6%)Education: Blocks AI training and AI search, 1 of 76 (1.3%)Education: Blocks AI training, allows AI search, 2 of 76 (2.6%)Education: No AI-specific rules, 71 of 76 (93.4%)93%Everything elseEverything else: Blocks all crawlers, Googlebot included, 305 of 6,644 (4.6%)Everything else: Blocks AI training and AI search, 428 of 6,644 (6.4%)6%Everything else: Blocks AI training, allows AI search, 831 of 6,644 (12.5%)13%Everything else: Blocks only AI search or user agents, 32 of 6,644 (0.5%)Everything else: Names AI agents, no full block, 233 of 6,644 (3.5%)Everything else: No AI-specific rules, 4,815 of 6,644 (72.5%)72%

Blocks all crawlers, Googlebot includedBlocks AI training and AI searchBlocks AI training, allows AI searchBlocks only AI search or user agentsNames AI agents, no full blockNo AI-specific rules

Each site counted once. “Blocks all crawlers” means Googlebot is disallowed from the whole site too, usually a wildcard rule on an app, API or login origin rather than an AI decision.

Block rates by segment

A reporter can quote any row. Percentages use the number of sites in the row as the base; “training only” means the site blocks at least one training agent and no AI search agent.

Segment Sites GPTBot OAI-SearchBot ClaudeBot Google-Extended Any training agent Training only llms.txt
All measured 8,404 16.7% 4.1% 15.2% 13.5% 21.1% 12.4% 8.5%
Top 1,000 863 20.3% 6.7% 18.9% 17.1% 26.4% 12.6% 8.3%
1,001 to 5,000 3,372 17.5% 4.2% 15.9% 14.0% 22.0% 13.0% 8.6%
5,001 to 10,000 4,169 15.4% 3.5% 13.8% 12.3% 19.3% 11.8% 8.3%
News publishers 768 34.8% 18.9% 32.0% 26.7% 47.7% 16.0% 13.2%
Retail and e-commerce 604 17.9% 4.5% 15.9% 13.2% 22.5% 12.6% 23.8%
Government 312 1.9% 0.0% 1.9% 2.2% 3.2% 2.2% 0.3%
Education 76 3.9% 0.0% 2.6% 3.9% 3.9% 2.6% 0.0%
Everything else 6,644 15.4% 2.6% 13.9% 12.6% 18.9% 12.5% 6.9%

By top-level domain

Country-code domains show local habits. Every TLD with at least 60 measured origins in the top 10,000:

TLD Sites GPTBot OAI-SearchBot ClaudeBot Google-Extended Any training agent Training only llms.txt
.com 3,876 15.4% 3.2% 14.2% 12.5% 20.5% 12.3% 10.1%
.net 400 17.2% 1.2% 16.2% 14.0% 22.2% 16.5% 7.3%
.jp 331 10.3% 5.7% 9.1% 8.5% 14.2% 6.3% 4.3%
.in 236 6.4% 0.4% 5.5% 5.9% 8.1% 5.9% 9.3%
.org 213 19.2% 1.9% 17.8% 16.0% 22.5% 15.0% 4.4%
.br 160 10.6% 3.1% 9.4% 7.5% 13.1% 6.9% 11.6%
.de 144 34.7% 8.3% 31.9% 27.1% 44.4% 18.8% 12.0%
.ru 143 7.0% 0.0% 4.2% 2.1% 9.1% 6.3% 5.2%
.fr 139 25.9% 10.8% 23.0% 23.0% 36.0% 14.4% 5.4%
.it 119 25.2% 9.2% 21.8% 22.7% 34.5% 14.3% 5.3%
.pl 114 28.9% 4.4% 14.0% 11.4% 39.5% 26.3% 5.3%
.es 89 28.1% 14.6% 23.6% 14.6% 31.5% 12.4% 8.1%
.tv 79 6.3% 1.3% 6.3% 6.3% 7.6% 6.3% 3.8%
.id 60 5.0% 0.0% 5.0% 5.0% 5.0% 5.0% 3.5%
.uk 60 31.7% 18.3% 26.7% 15.0% 33.3% 11.7% 11.8%

Every agent, with what its vendor says it does

Classes come from each vendor’s own crawler page, quoted below. Block shares are of all 8,404 measured sites. “By name” counts sites that list the token in a group that disallows the whole site; “any rule” adds sites that block it through a wildcard group.

User agent Vendor Class Sites blocking by name By name Any rule Partly blocked
GPTBot OpenAI Training 1,407 16.7% 21.4% 2.1%
ClaudeBot Anthropic Training 1,275 15.2% 20.0% 1.8%
Google-Extended Google Training 1,133 13.5% 18.2% 1.5%
Applebot-Extended Apple Training 1,052 12.5% 17.4% 0.8%
Meta-ExternalAgent Meta Training 1,079 12.8% 17.8% 1.1%
Amazonbot Amazon Training 1,125 13.4% 18.4% 1.0%
CCBot Common Crawl Training 1,399 16.6% 21.6% 1.0%
OAI-SearchBot OpenAI AI search 345 4.1% 8.9% 2.2%
Claude-SearchBot Anthropic AI search 271 3.2% 8.2% 1.3%
Meta-WebIndexer Meta AI search 152 1.8% 7.1% 0.5%
Amzn-SearchBot Amazon AI search 101 1.2% 6.5% 0.1%
PerplexityBot Perplexity AI search 574 6.8% 11.5% 2.1%
DuckAssistBot DuckDuckGo AI search 239 2.8% 7.9% 0.8%
ChatGPT-User OpenAI User fetch 516 6.1% 11.1% 2.2%
Claude-User Anthropic User fetch 290 3.5% 8.5% 1.2%
Meta-ExternalFetcher Meta User fetch 283 3.4% 8.6% 0.5%
Amzn-User Amazon User fetch 127 1.5% 6.9% 0.1%
Perplexity-User Perplexity User fetch 297 3.5% 8.7% 1.1%
MistralAI-User Mistral User fetch 209 2.5% 7.8% 0.3%
anthropic-ai Anthropic Legacy token 1,090 13.0% 18.1% 0.8%
Claude-Web Anthropic Legacy token 955 11.4% 16.6% 0.4%
Bytespider ByteDance Undocumented 1,328 15.8% 21.0% 0.9%
Googlebot Google Search engine 23 0.3% 3.9% 5.0%
Bingbot Microsoft Search engine 85 1.0% 5.2% 3.9%
Applebot Apple Search engine 118 1.4% 6.0% 1.5%
User agent Vendor page Class Vendor’s own words (accessed 11 Nov 2026)
GPTBot OpenAI Training “crawl content that may be used in training our generative AI foundation models”
ClaudeBot Anthropic Training “collecting web content that could potentially contribute to their training”
Google-Extended Google Training “manage whether content Google crawls from their sites may be used for training future generations of Gemini models … and for grounding”
Applebot-Extended Apple Training “opt-out from having their content used to train generative foundation models by disallowing Applebot-Extended”
Meta-ExternalAgent Meta Training “crawls the web for use cases such as training foundation AI models or improving products by indexing content directly”
Amazonbot Amazon Training “may be used to train Amazon AI models”
CCBot Common Crawl Training “an open repository of web crawl data that is universally accessible and analyzable by anyone”
OAI-SearchBot OpenAI AI search “used to surface websites in search results in ChatGPT's search features”
Claude-SearchBot Anthropic AI search “navigates the web to improve search result quality for users”
Meta-WebIndexer Meta AI search “navigates the web to improve Meta AI search result quality for users”
Amzn-SearchBot Amazon AI search “does not crawl content for generative AI model training”
PerplexityBot Perplexity AI search “designed to surface and link websites in search results on Perplexity. It is not used to crawl content for AI foundation models”
DuckAssistBot DuckDuckGo AI search “crawls pages in real-time for our AI-assisted answers … This data is not used in any way to train AI models”
ChatGPT-User OpenAI User fetch “not used for crawling the web in an automatic fashion. Because these actions are initiated by a user, robots.txt rules may not apply”
Claude-User Anthropic User fetch “When individuals ask questions to Claude, it may access websites using a Claude-User agent”
Meta-ExternalFetcher Meta User fetch “fetches individual links at a user's request … may bypass robots.txt”
Amzn-User Amazon User fetch “supports user actions, such as responding to Alexa queries … may not follow all robots.txt directives”
Perplexity-User Perplexity User fetch “supports user actions within Perplexity … not used for web crawling or to collect content for training”
MistralAI-User Mistral User fetch “for user actions in Vibe … not used for crawling the web in any automatic fashion, nor to crawl content for generative AI training”
anthropic-ai Anthropic Legacy token “Not listed on Anthropic's current crawler page; an older token many robots.txt files still name”
Claude-Web Anthropic Legacy token “Not listed on Anthropic's current crawler page; an older token many robots.txt files still name”
Bytespider ByteDance Undocumented No vendor documentation page could be retrieved; classed separately

Google-Extended and Applebot-Extended are control tokens, not separate crawlers: Google says Google-Extended “doesn’t have a separate HTTP request user agent string” and does not affect inclusion in Google Search. Google also says it intends to support the AIPREF training-preference signal (user-agent: __usage_ai-training) in early 2027.

Look up a top-1,000 site

All 248 origins in the CrUX top 1,000 band (adult sites left out) whose robots.txt names at least one AI agent. “Training blocked” counts the seven training agents fully blocked by any rule.

Origin Category Training blocked (of 7) GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Claude-SearchBot Google-Extended PerplexityBot llms.txt

llms.txt adoption

llms.txt is a proposal, published by Jeremy Howard in September 2024, for a Markdown file at /llms.txt that helps AI agents use a site. We counted a file as valid only if /llms.txt returned HTTP 200 as non-HTML text at the same path and started with an H1 line, the one section the proposal requires. 671 of 7,932 checked sites (8.5%) qualified. The policy split is sharper than the rank split: sites that name AI agents in robots.txt without fully blocking any publish llms.txt at 39.0%, against 7.1% for sites with no AI-specific rules and 5.3% for sites that block GPTBot by name.

Top 1,000Top 1,000: 66 of 793 checked sites (8.3%)8.3% (66)1,001 to 5,0001,001 to 5,000: 276 of 3,194 checked sites (8.6%)8.6% (276)5,001 to 10,0005,001 to 10,000: 329 of 3,945 checked sites (8.3%)8.3% (329)News publishersNews publishers: 96 of 729 checked sites (13.2%)13.2% (96)Retail and e-commerceRetail and e-commerce: 142 of 596 checked sites (23.8%)23.8% (142)GovernmentGovernment: 1 of 291 checked sites (0.3%)0.3% (1)EducationEducation: 0 of 73 checked sites (0.0%)0.0% (0)Everything elseEverything else: 432 of 6,243 checked sites (6.9%)6.9% (432)Blocks training and searchBlocks training and search: 22 of 596 checked sites (3.7%)3.7% (22)Blocks training onlyBlocks training only: 88 of 1,037 checked sites (8.5%)8.5% (88)Names AI, no full blockNames AI, no full block: 143 of 367 checked sites (39.0%)39.0% (143)No AI-specific rulesNo AI-specific rules: 417 of 5,873 checked sites (7.1%)7.1% (417)

Base: sites where robots.txt allowed our crawler to request /llms.txt and we requested it.

A robots.txt that blocks GPTBot, explained

These two examples follow the vendors' documented tokens. They are not recommendations: which one fits depends on whether a site wants AI search referrals, and on its licensing position. Grouping several User-agent lines above one rule set is valid under RFC 9309, which says matching groups' rules "MUST be combined."

# Opt out of AI model training, stay visible in AI search and assistants
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: Amazonbot
User-agent: CCBot
Disallow: /

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

The first file asks training agents to stay out while leaving the search agents of OpenAI, Anthropic and Perplexity allowed. Google-Extended blocks Gemini training and grounding without touching Googlebot, so Google Search inclusion is unchanged. 1,039 sites in our sample already run a policy of this shape.

# Opt out of AI training AND AI search answers (classic search unaffected)
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: Amazonbot
User-agent: CCBot
Disallow: /

The second also removes the site from ChatGPT, Claude and Perplexity search answers. OpenAI notes that sites opted out of OAI-SearchBot "can still appear as navigational links." User-fetch agents are listed, but OpenAI, Meta and Amazon all say user-initiated fetchers may not follow robots.txt.

robots.txt is a request, not enforcement

RFC 9309, the IETF standard for robots.txt published in September 2022, says the rules "are not a form of access authorization," and its security section says the protocol "is not a substitute for valid content security measures." A Disallow line only works on crawlers that choose to honor it. Sites that need enforcement add user-agent and IP verification at a firewall or CDN, using the IP ranges OpenAI, Perplexity and Amazon publish, or Common Crawl's reverse DNS names. Our numbers measure stated policy, not what any crawler actually did.

What this means for MSPs and IT teams

For clients who publish content, the decision is now two decisions: training and search. Block lists copied between sites often put search agents such as OAI-SearchBot or PerplexityBot in the same group as GPTBot, which opts the site out of AI search as well; 25.9% of GPTBot blockers in the top 10,000 block OpenAI's search agent too. Check three things in a client's robots.txt: whether training and search tokens are in separate groups, whether a wildcard Disallow: / on a marketing origin blocks every agent by accident (3.9% of top origins block Googlebot as well), and whether legacy tokens such as anthropic-ai and Claude-Web are the only Anthropic entries, since Anthropic's current page lists ClaudeBot, Claude-SearchBot and Claude-User. Where clients need real enforcement, robots.txt has to be paired with bot rules at the edge. For context on how much traffic ChatGPT carries, see our ChatGPT market share data; for the state laws that now touch AI use, see the state AI laws tracker. The same passive method underpins our DMARC adoption statistics.

Frequently asked questions

How many websites block GPTBot?

In our November 2026 check of the 8,404 most visited origins in Chrome's CrUX list that returned a usable robots.txt answer, 1,407 (16.7%) fully block GPTBot by name. Counting sites whose robots.txt blocks every crawler with a wildcard rule, 1,802 (21.4%) are closed to GPTBot.

Do sites that block GPTBot also block ChatGPT search?

Mostly not. Of the 1,407 sites that block GPTBot by name, 365 (25.9%) also block OAI-SearchBot, the agent OpenAI says decides whether a site can appear in ChatGPT search answers. 537 (38.2%) also block ChatGPT-User.

Which AI crawler is blocked the most?

By name, GPTBot (16.7% of sites), CCBot (16.6%) and Bytespider (15.8%) lead. The search agents trail: PerplexityBot 6.8%, OAI-SearchBot 4.1%, Claude-SearchBot 3.2%.

How many websites have an llms.txt file?

671 of the 7,932 top sites we could check (8.5%) serve a valid /llms.txt that starts with a Markdown H1, as the llmstxt.org proposal requires. Among sites that block GPTBot by name the rate is 5.3%.

Does blocking GPTBot in robots.txt stop OpenAI?

robots.txt is a request, not access control: RFC 9309 says its rules "are not a form of access authorization." OpenAI says disallowing GPTBot signals content should not be used for training, while ChatGPT-User fetches, started by users, may not follow robots.txt rules.

How do I block GPTBot but stay in ChatGPT search?

Add a group with User-agent: GPTBot and Disallow: /, and leave OAI-SearchBot allowed (no group, or Allow: /). OpenAI documents the two settings as independent. The copyable example on this page shows the same split for Anthropic, Google, Apple, Meta and Amazon.

Download the data. ai-crawler-robots-txt-blocks-crux-top-10000-nov-2026.csv (CSV). Licensed under CC BY 4.0: free to reuse with attribution to The Uptime Report.

Free to cite and republish with a link to this page. The CSV holds the aggregate block rates for every agent by rank band, category and TLD; it contains no per-site rows.

Methodology

Site list. Chrome UX Report (CrUX) global popularity ranks for the October 2026 release (yyyymm 202610), all origins in the 1,000, 5,000 and 10,000 rank buckets: 10,000 origins. CrUX ranks origins by navigations on a log10 scale with half steps, and each bucket excludes the one above it, so the bands are top 1,000, 1,001 to 5,000 and 5,001 to 10,000; order within a bucket is not published. We used the monthly CSV export of the public BigQuery table published in the zakird/crux-top-lists repository on GitHub (data/global/202610.csv.gz, committed 10 November 2026), so no BigQuery account was needed. Google states that "CrUX datasets by Google are licensed under a Creative Commons Attribution 4.0 International License." Units are origins, so one company can appear several times (for example country versions of the same store).

Fetching. On 11 November 2026 we requested /robots.txt once for every origin, following up to five redirects as RFC 9309 asks, with the user agent "UptimeReportResearch/1.0 (+https://theuptimereport.com/about/)". Where that robots.txt allowed our own agent to fetch /llms.txt, we requested /llms.txt as a second check, at least 1.2 seconds later. Requests were paced to about 7 per second across all sites, at most two per site. No other paths were requested.

Coverage. 7,149 origins returned a robots.txt file, 1,255 had none (a 4xx response or an HTML page, which means all agents are allowed), 1,101 refused us (HTTP 401, 403, 429, a 5xx, or a bot-challenge page), and 495 failed at the network level. The 1,596 refused or failed origins are excluded, leaving 8,404 measured. Refusals were not worked around.

Parsing and classes. Our parser follows RFC 9309: user-agent groups matched case-insensitively on the product token, matching groups merged, fallback to the * group, longest matching rule wins with Allow winning ties, * and $ wildcards. An agent is "fully blocked" if three probe paths (a news-style article path, a product path and /about) are all disallowed, whether or not the homepage itself is left open with an Allow: /$ line; "by name" if the rule comes from a group naming that agent, "any rule" if it may come from the * group. "Partly blocked" means a named group disallows some paths but not all three. Training, search and user-fetch classes follow each vendor's documentation; CCBot is grouped with training agents because it builds a bulk open corpus rather than serving search or user requests, Bytespider has no retrievable vendor page and is shown separately, and anthropic-ai and Claude-Web are legacy tokens.

Category rule. Government: host ends in .gov or .mil, or a gov, gob, gouv, go, govt, gv, gub or mil label sits before a country code, or europa.eu, gov.uk, canada.ca, bund.de, admin.ch. Education: .edu, or edu or ac before a country code. News publishers: robots.txt lists a Sitemap URL containing "news" (the Google News sitemap convention). Retail and e-commerce: robots.txt disallows a path containing cart, checkout, basket or wishlist. Rules apply in that order; everything else is "Everything else". The rule is reproducible but approximate.

Limitations. robots.txt states a request; it does not show what crawlers do, and many sites block AI agents at the CDN instead, which this method cannot see. A site could serve a different robots.txt to a real AI crawler than to us. Origins that refused our agent may differ from those that answered. This is a single snapshot.

Not advice. This is a measurement of public robots.txt files, not legal or SEO advice for any site.

Cite as:The Uptime Report. "16.7% of the Top 10,000 Websites Block GPTBot by Name, but Only 4.1% Block ChatGPT Search." November 2026. https://theuptimereport.com/block-gptbot-top-websites-2026/

References

  1. Chrome UX Report methodology, including the CC BY 4.0 dataset licence. Accessed November 2026.
  2. CrUX metrics: popularity rank definition. Accessed November 2026.
  3. zakird/crux-top-lists, monthly CrUX top-origin exports (202610.csv.gz). Accessed November 2026.
  4. RFC 9309, Robots Exclusion Protocol. Accessed November 2026.
  5. OpenAI, Overview of OpenAI crawlers. Accessed November 2026.
  6. Anthropic, crawler and fetcher documentation. Accessed November 2026.
  7. Google, common crawlers and Google-Extended. Accessed November 2026.
  8. Apple, About Applebot. Accessed November 2026.
  9. Meta, web crawlers. Accessed November 2026.
  10. Amazon, Amazon web search crawlers. Accessed November 2026.
  11. Perplexity, crawlers. Accessed November 2026.
  12. Common Crawl, CCBot. Accessed November 2026.
  13. DuckDuckGo, DuckAssistBot. Accessed November 2026.
  14. Mistral AI, robots. Accessed November 2026.
  15. llmstxt.org, the /llms.txt proposal. Accessed November 2026.

Research library

More original data reports from The Uptime Report. Every report includes its methodology and a free CSV download. Browse all research

Threats and breaches

Infrastructure and uptime

Regulation and disclosure

Workforce and pay

MSP market and spending

Comments

One response to “16.7% of the Top 10,000 Websites Block GPTBot by Name, but Only 4.1% Block ChatGPT Search”

  1. […] Block GPTBotAI crawler blocks across top 10k sites […]

Leave a Reply

Your email address will not be published. Required fields are marked *