The AI crawlers that matter in 2026 fall into three functional types: training crawlers like GPTBot, ClaudeBot, and Google-Extended that feed model training, search crawlers like OAI-SearchBot and Claude-SearchBot that build retrieval indexes, and user-triggered agents like ChatGPT-User and Perplexity-User that fetch pages live when someone asks a question. The distinction is the whole game: blocking a training bot costs you nothing in citations, while blocking a search or user agent removes you from AI answers entirely. OpenAI alone now runs three separate user agents, and Cloudflare traffic data has consistently shown GPTBot among the highest-volume AI crawlers on the web, so getting the list right in robots.txt is a visibility decision, not a housekeeping one.
What are the three types of AI crawler?
Every bot on this list does one of three jobs, and the job determines whether you should ever block it.
Training crawlers harvest content for future model training runs. Their visits produce no immediate citations; they shape what next year’s models know. GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta’s crawler, and CCBot live here.
Search index crawlers build the retrieval indexes AI answers are assembled from. OAI-SearchBot populates ChatGPT search, Claude-SearchBot supports Claude’s web answers, PerplexityBot feeds Perplexity’s index. Block these and your pages cannot be retrieved, which means they cannot be cited.
User-triggered fetchers act on behalf of a live user: ChatGPT-User fires when someone in ChatGPT opens or browses your page, Claude-User when Claude does, Perplexity-User when a Perplexity answer needs your live page. These are closest to a human visit, and several providers state user-triggered fetchers may not strictly follow robots.txt because a person initiated the request.
The naming pattern is now an industry convention: each major operator runs a plain bot for training, a SearchBot for indexing, and a User agent for live fetches. Once you read a user agent string through that lens, an unfamiliar bot usually classifies itself.
The strategic default for any business that wants AI visibility: allow all three types, or at minimum allow every search and user agent, the decision framework we laid out in should you block AI crawlers.
Not sure which bots your robots.txt is currently blocking, or what it is costing you? Get a free AI visibility audit and we will check your crawl access alongside your actual citations across every major engine.
Which user agents does each AI company run?
The full reference list, by operator:
1. OpenAI
GPTBot (training), OAI-SearchBot (ChatGPT search index), ChatGPT-User (live browsing on behalf of users). Robots.txt tokens match those names. The pair that controls citations is OAI-SearchBot plus ChatGPT-User; GPTBot only affects training data. All three publish IP ranges for verification.
2. Anthropic
ClaudeBot (training), Claude-SearchBot (search indexing), Claude-User (user-triggered fetches). Same three-way split as OpenAI. Claude’s live answers also lean on the Brave Search index, so Brave visibility feeds Claude citations indirectly.
3. Google
Googlebot (classic search, also feeds AI Overviews and AI Mode), Google-Extended (a robots.txt token controlling Gemini and Vertex AI training, not a separate crawler), Google-CloudVertexBot (Vertex customer crawling). Note the trap: you cannot block AI Overviews without blocking Google Search itself, because both run through Googlebot. Google-Extended only opts you out of training.
4. Perplexity
PerplexityBot (index) and Perplexity-User (live fetches). Perplexity retrieves in real time on nearly every query, so crawl access converts to citations faster here than anywhere else, the dynamic behind the playbook in how to rank in Perplexity AI.
5. Microsoft
Bingbot does triple duty: Bing search, Copilot answers, and the index behind ChatGPT search. Blocking Bingbot removes you from three surfaces at once, which is why Bing Webmaster Tools and IndexNow sit at the center of how to get indexed on Bing.
6. Everyone else
Applebot and Applebot-Extended (Apple search features and AI training opt-out), Amazonbot (Alexa and Rufus), meta-externalagent and FacebookBot (Meta AI), LinkedInBot (link previews and Microsoft surfaces), Bytespider (ByteDance, historically the least compliant major crawler), CCBot (Common Crawl, the open dataset many models train on), DuckAssistBot (DuckDuckGo AI answers), and MistralAI-User (Mistral’s Le Chat fetches).
Which crawlers should you allow, and which can you safely block?
Decision table by goal:
- Maximize AI citations (most businesses). Allow everything above. Every retrieval and user agent blocked is an answer surface you exit. Training bots cost bandwidth but build brand presence inside future models, the long-game asset we described in how to get your brand into LLM training data.
- Protect proprietary content while keeping citations. Block training bots (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider) and allow search plus user agents (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Bingbot, Googlebot). You stay citable today without donating training data.
- Full opt-out (publishers negotiating licensing deals). Block everything except Googlebot and Bingbot if you still want classic search traffic. Understand the cost: you disappear from ChatGPT, Claude, Perplexity, and Copilot answers entirely.
One nuance worth pricing in before you choose: the crawl-to-referral ratio varies enormously by operator. Cloudflare’s crawl-versus-refer data has shown AI platforms crawling hundreds to thousands of pages for every visitor they send back, with training-heavy crawlers at the extreme end. That asymmetry is the honest argument for the selective strategy: retrieval and user agents at least sit in the citation path, while training bots consume bandwidth against a payoff measured in years. Sites on metered hosting or with tens of millions of URLs should also set crawl-delay expectations per bot and watch aggregate AI crawl load, which for large publishers now rivals classic search engine traffic.
Verification matters as much as policy. Fake bots spoof these user agent strings constantly, so confirm real crawlers by reverse DNS or published IP ranges before whitelisting anything at the firewall or CDN level. Cloudflare, Fastly, and Akamai all now ship managed rules for verified AI bots.
What does a correct robots.txt look like for each strategy?
The syntax is simple; the mistakes come from copying stale templates. For the maximize-citations default, you need no AI-specific rules at all: crawlers are allowed unless disallowed, so the only work is auditing for leftover blocks. Thousands of sites still carry a blanket GPTBot disallow added during the 2023 backlash that now also signals a closed door to an ecosystem that did not exist when the rule was written.
For the protect-content-keep-citations strategy, the pattern is explicit pairs: User-agent: GPTBot with Disallow: /, repeated for ClaudeBot, Google-Extended, Applebot-Extended, CCBot, and Bytespider, while OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Bingbot, and Googlebot get no disallow at all. Order matters less than completeness; missing one training token quietly reopens the door, and blocking one retrieval token quietly closes a citation surface.
Three implementation details save real pain. First, robots.txt matches the most specific user-agent block, so a User-agent: * disallow does not exempt bots you name elsewhere; test with each operator’s published token. Second, changes take effect only after the next crawl of robots.txt itself, which can lag days for low-traffic sites, so do not judge a change in 24 hours. Third, pair the file with your CDN settings: Cloudflare’s AI bot controls can override robots.txt intent with one dashboard toggle, and mismatched layers are the most common cause of “we allowed the bot but it never came.”
How do you check what is crawling you right now?
Three steps, thirty minutes, and worth repeating monthly because the answer changes. New agents appear without announcement, operators rotate IP ranges, and a CMS migration or CDN rule change can silently cut off bots you meant to allow. First, pull 30 days of server or CDN logs and filter user agent strings against the list above; Cloudflare’s bot analytics and its AI Audit tool surface this without log work. Second, compare hits against your robots.txt to find bots ignoring your directives; repeat offenders get IP-level blocks. Third, check which pages the search and user agents actually fetch, because those pages are your live citation surface, and JavaScript-heavy pages they cannot parse are wasted visits, the rendering problem we detailed in can AI crawlers read JavaScript. Most AI fetchers read raw HTML only, so anything client-rendered is invisible regardless of crawl permissions.
FAQ: AI crawlers and user agents
What is the difference between GPTBot and ChatGPT-User?
GPTBot is OpenAI’s training crawler; it collects content for future model training and never produces a citation. ChatGPT-User is a live fetcher that visits your page when a real ChatGPT user browses or asks about it, and OAI-SearchBot builds the search index between them. Blocking GPTBot preserves your citations; blocking OAI-SearchBot or ChatGPT-User removes you from ChatGPT answers.
Does blocking AI crawlers stop AI engines from mentioning my brand?
No. Engines can still mention your brand from training data, third-party coverage, Reddit threads, and directories even if they cannot crawl your site. Blocking crawlers only removes your own pages as sources, which usually means the AI describes you using everyone else’s words. For most brands that is the worst of both worlds.
Which AI crawlers ignore robots.txt?
Bytespider has the worst documented compliance record among major crawlers, and user-triggered fetchers like ChatGPT-User and Perplexity-User may fetch pages despite robots.txt because a human initiated the request; Perplexity’s fetching practices drew a public Cloudflare report in 2024. For hard enforcement, robots.txt is a request; CDN-level verified-bot rules are a wall.
Do AI crawlers execute JavaScript?
Mostly no. Googlebot renders JavaScript, but GPTBot, ClaudeBot, PerplexityBot, and the user-triggered fetchers read raw HTML and skip client-side rendering. If your content only exists after JavaScript runs, it does not exist for AI retrieval. Server-side rendering or prerendering is the fix, and it is one of the highest-ROI technical changes in GEO.
How do I verify a crawler is really who it claims to be?
Check the source IP against the operator’s published ranges, OpenAI, Anthropic, Google, Microsoft, and Perplexity all publish theirs, or run reverse DNS lookup on the requesting IP. User agent strings are trivially spoofed by scrapers, so never grant firewall exceptions on the string alone. CDN verified-bot programs from Cloudflare and Fastly handle this automatically.
How often should I review my AI crawler policy?
Quarterly, minimum. New agents appear constantly, DuckAssistBot, MistralAI-User, and Claude-SearchBot all rolled out within roughly a year of each other, and operators add tokens or change behavior with little announcement. Put the robots.txt review on the same calendar as your citation tracking so access changes and visibility changes stay connected.
The bottom line: the AI crawler list is really three lists, training, search, and user-triggered, and your robots.txt should treat them differently on purpose. Know which token controls which surface, allow the agents that produce citations, verify the traffic that claims to be them, and revisit quarterly as new bots ship. The businesses losing AI visibility to a copy-pasted block-everything robots.txt from 2023 outnumber the ones losing anything to the crawlers themselves.
Want a crawl-access check bundled with a real citation baseline? Run your free AI visibility audit and see whether the engines can reach you, and what they say when they do.
Tagged