Should You Block AI Crawlers? robots.txt Guide 2026
Decide which AI bots to allow in robots.txt: block GPTBot and ClaudeBot training while keeping OAI-SearchBot and PerplexityBot so AI search can reach you.
If you want AI assistants to recommend you, block AI training crawlers and allow AI search and retrieval crawlers. Vendors now split these jobs across separate user agents, so GPTBot and OAI-SearchBotcan get different answers in the same robots.txt file. A blanket block on every "AI" bot usually costs you visibility without stopping the agents that ignore robots.txt.
Research record
Each token below was checked against its operator's documentation on November 7, 2026. Vendors rename bots often, so recheck before editing a production file. Product behavior is taken from the scanner source code.
What is the difference between AI training crawlers and AI search crawlers?
A training crawler collects pages that may go into a future model. A search crawler indexes pages so an AI product can find, quote and link them when someone asks a question. A user-initiated fetcher reads one page because a person asked the assistant to. Blocking a training crawler states a content-use preference; blocking a search crawler or fetcher directly reduces where your pages can appear.
The OpenAI crawler documentation says each setting is independent, so you can allow OAI-SearchBot while disallowing GPTBot. It adds that if you allow both, OpenAI may use one crawl for both purposes.
Which AI crawler user agents should you know in 2026?
Twelve tokens cover the main decisions: three from OpenAI, three from Anthropic, two each from Perplexity and Apple, one from Google and Common Crawl's CCBot. Two are control tokens with no crawler behind them; two are user fetchers that may skip robots.txt. The last column shows what a Disallow changes.
| Token | Operator | Purpose | Effect of blocking it |
|---|---|---|---|
| GPTBot | OpenAI | Collects content that may train OpenAI models | Training opt-out. Does not remove you from ChatGPT search. |
| OAI-SearchBot | OpenAI | Surfaces sites in ChatGPT search | Keeps pages out of ChatGPT search answers; OpenAI says they can still appear as navigational links. |
| ChatGPT-User | OpenAI | Fetches a page a user or Custom GPT asks for | Weak opt-out: OpenAI says robots.txt rules may not apply. |
| ClaudeBot | Anthropic | Collects content that could train Anthropic models | Excludes future content from training datasets. |
| Claude-SearchBot | Anthropic | Indexes content for Claude search results | May reduce your visibility in Claude search results. |
| Claude-User | Anthropic | Retrieves a page for a Claude user query | Stops those fetches; Anthropic says its bots honor robots.txt. |
| Google-Extended | Control token, no crawler: Gemini training and grounding | No effect on Google Search inclusion, ranking or AI Overviews. | |
| PerplexityBot | Perplexity | Surfaces and links sites in Perplexity; not used for model training | Reduces links in Perplexity answers. Opts out of nothing training-related. |
| Perplexity-User | Perplexity | Visits a page during a user request | Weak opt-out: Perplexity says it generally ignores robots.txt. |
| Applebot | Apple | Powers Spotlight, Siri and Safari search; data may also train Apple models | Blocks the crawler behind Apple search features. |
| Applebot-Extended | Apple | Control token, does not crawl: Apple model training | Training opt-out; Apple says pages can still appear in its search results. |
| CCBot | Common Crawl | Builds the open Common Crawl web archive | Asks Common Crawl not to fetch your pages; it checks robots.txt first. |
Does blocking Google-Extended remove you from AI Overviews?
No. Google-Extended governs whether content Google crawls can train future Gemini models and ground answers in Gemini Apps and Vertex AI. Google says it does not affect inclusion in Google Search and is not a ranking signal. AI Overviews and AI Mode are Search features, so Googlebot access and snippet rules control them instead.
The Google-Extended entry in Google's crawler list confirms it has no separate user agent, so it never appears in your logs. To limit AI features, Google's AI features guidance points to nosnippet, data-nosnippet, max-snippet or noindex. The robots meta tag reference says nosnippet stops content being a direct input for AI Overviews and AI Mode, but it also removes your normal search snippets.
Should you block AI crawlers? A decision framework
Decide by business model. If AI answers send you customers, block training and allow search and retrieval. If revenue depends on people reading your original text, block more and accept lower AI visibility. If you want maximum reach and accept training use, allow everything. Revisit the choice when vendors change their bots.
| Policy | Fits | You gain | You give up |
|---|---|---|---|
| Block training, allow search | Local businesses, SaaS, ecommerce, most B2B sites | A training opt-out while staying reachable for ChatGPT search, Claude, Perplexity and Google | Little; the opt-out depends on vendors honoring it |
| Block all AI agents | Publishers, paywalled research, original datasets | The strongest signal you can send to compliant AI bots | Citations and links in AI answers; Google AI Overviews still draw on Googlebot unless you limit snippets |
| Allow all | Documentation, open-source projects, reach-first brands | No accidental blocks; simplest file | Any say over training use |
Access is only the entry ticket. If ChatGPT still skips you after access is open, look at entity clarity and proof, covered in why ChatGPT does not mention your business.
What robots.txt rules should you copy?
Pick one template and place the file at the root of each host; rules apply only to the host, protocol and port that serve them. A group may stack several User-agent lines before its rules, and token matching is case-insensitive under RFC 9309, the Robots Exclusion Protocol.
Template 1: block training, allow search and retrieval
# AI training opt-outs
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /
# Everyone else, including OAI-SearchBot,
# Claude-SearchBot and PerplexityBot
User-agent: *
Disallow: /admin/
Disallow: /checkout/
Sitemap: https://www.example.com/sitemap.xmlTemplate 2: block AI agents, keep Google and Bing search
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /
User-agent: *
Disallow: /admin/
Sitemap: https://www.example.com/sitemap.xmlChatGPT-User and Perplexity-User are listed for completeness only; enforce with a server or CDN rule.
Template 3: allow all
User-agent: *
Disallow: /admin/
Sitemap: https://www.example.com/sitemap.xmlWatch the group-selection trap
A crawler obeys only the most specific group naming it. Google's robots.txt specification says specific groups and the * group are never combined. Add a group for OAI-SearchBot and it stops reading your wildcard Disallow: /admin/, so repeat private paths inside that group. The same spec says paths are case-sensitive, Google ignores Crawl-delay, and content past 500 KiB is ignored.
Is robots.txt enough to stop AI bots?
No. RFC 9309 states that robots.txt rules are not a form of access authorization. Reputable crawlers comply voluntarily, some user fetchers do not, and scrapers can fake any user agent. State your preference in robots.txt, then enforce it with login, server rules or a CDN or WAF policy.
Cloudflare's AI Crawler and Agent access policies(Security > Settings, filtered by Bot traffic) sort bots into Search, Agent and Training, each set to Block, Block on pages with ads, or Allow. Cloudflare says Training actions also apply to mixed-purpose crawlers used for both training and search, so a Training block can catch a search crawler you meant to keep.
Check your own defaults. That page, last updated November 2, 2026, says new domains default to Allow for all three categories, while a July 1, 2026 Cloudflare changelog had announced that new domains from September 15, 2026 would block Training and Agent bots on pages with ads. Its managed robots.txt can also prepend a Content-signalline that Search Console may flag as "Syntax not understood".
If an edge rule starts rejecting Google's fetch of robots.txt itself, work through why Search Console says robots.txt is unreachable. Anthropic also warns that blocking its IP addresses may not work as an opt-out, because the crawler can no longer read your robots.txt.
How do you check which AI bots actually visit your site?
Read your server or CDN logs, then confirm each claimed bot comes from its operator. A user-agent string proves nothing alone; Common Crawl warns about crawlers falsely claiming to be CCBot. Most operators publish reverse DNS hostnames or IP range files to check against.
- Export a week of access logs. Filter on the tokens above and note paths and status codes.
- Compare status codes with your intent. An allowed bot should get 200s on public pages; a 403 points to a WAF or CDN rule, not robots.txt.
- Verify by reverse DNS where supported. Google's Googlebot verification steps run
hoston the IP, expect a googlebot.com, google.com or googleusercontent.com name, then resolve it back to the same IP. Applebot uses applebot.apple.com and CCBot uses crawl.commoncrawl.org. - Match IPs against published ranges. OpenAI publishes gptbot.json, searchbot.json and chatgpt-user.json; Perplexity publishes perplexitybot.json and perplexity-user.json; Anthropic, Apple, Google and Common Crawl publish lists too.
- Retest after a day. OpenAI says search changes take about 24 hours and Perplexity says up to 24 hours.
Signed requests are a newer option: Cloudflare's Web Bot Auth documentation describes bots that sign each request with an Ed25519 key you can look up, and notes that a bot must also register to be listed as verified.
What does VisRank check about AI crawler access?
VisRank's AEO checks evaluate your robots.txt for the exact scanned path against a list of named crawlers, using most-specific-group matching. The check fails only when a search crawler is blocked or Google snippets are disabled. Training opt-outs such as GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot are reported as preferences and do not count against you.
- Search crawlers: Googlebot, bingbot, OAI-SearchBot, Claude-SearchBot, PerplexityBot, Applebot and Amzn-SearchBot. Blocking any of them raises a high-severity "Search or AI answer access is restricted" issue.
- Snippet controls:
nosnippetormax-snippet:0triggers the same issue. - User fetchers: blocked ChatGPT-User, Claude-User or Perplexity-User rules are listed separately.
- Unreadable files: if robots.txt times out or returns HTML, the check is marked skipped instead of passing.
The AI Search Readiness Checker shows this finding with the other AEO issues. VisRank's own crawler identifies as VisRankBot/1.0 and signs requests with Web Bot Auth, publishing its Ed25519 public key at /.well-known/http-message-signatures-directory. If signing fails, the request goes out unsigned rather than blocking the scan. Automatic page discovery follows VisRankBot or wildcard rules, as the VisRankBot crawler page explains.
Does llms.txt help control AI crawlers?
No. llms.txt is a proposal published at llmstxt.org for a Markdown file that helps language models and agents use a site. It grants or denies nothing. Google's generative AI optimization guide says Google Search ignores it, so it neither helps nor harms Google visibility.
Other tools may still read it, so the scanner reports whether one exists and offers an llms.txt checker. Treat it as optional once crawl access, indexability and clear content are sorted; the guide to optimizing for AI search covers that order of work.
Frequently asked questions about blocking AI crawlers
Will blocking GPTBot remove my site from ChatGPT search?
No. OpenAI uses GPTBot for training and OAI-SearchBot for ChatGPT search, and each setting is independent. Keep OAI-SearchBot allowed.
Does Google-Extended affect Google rankings or AI Overviews?
No. Google says Google-Extended does not affect inclusion or ranking in Google Search. AI Overviews and AI Mode rely on Googlebot; limit them with nosnippet, max-snippet, data-nosnippet or noindex.
Do AI user agents like ChatGPT-User obey robots.txt?
Not reliably. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them. Anthropic says its bots, including Claude-User, honor robots.txt.
Is robots.txt enough to keep AI bots off private content?
No. RFC 9309 says robots.txt is not access authorization. Use login, server, CDN or WAF rules for anything that must stay private.
Does an llms.txt file block or allow AI crawlers?
No. llms.txt is a proposal for describing a site to language models, not an access control. Google says Google Search ignores it.
Key takeaways
- Training and search crawlers use different tokens, so you can opt out of training without leaving AI search.
- GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot are the main training or dataset tokens to consider blocking.
- OAI-SearchBot, Claude-SearchBot and PerplexityBot decide whether those products can index and link your pages.
- Google-Extended does not touch Search rankings or AI Overviews; snippet controls do.
- A specific user-agent group replaces the wildcard group, so repeat private-path rules inside it.
- robots.txt is voluntary; enforce real restrictions at the server, CDN or login layer and verify bots by DNS or published IP ranges.
Want to know whether your robots.txt blocks a search crawler on your important pages? Run a free VisRank scan →
Sources checked on November 7, 2026: OpenAI crawlers, Anthropic crawlers, Google common crawlers, Google AI features, Perplexity crawlers, Applebot, CCBot, Common Crawl FAQ, RFC 9309, Cloudflare managed robots.txt.