Type curl -A "GPTBot" https://yoursite.com into a terminal and you get exactly the same page a human visitor gets. Most sites have no idea how often that happens, or who is actually asking.
Search engines used to be simple: Googlebot and Bingbot crawled, you optimized for both, and that was the whole game. Now there are more than a dozen named crawlers run by AI companies, each with a different job. Some train models, some power live answers, some do both. They do not behave the same way, and they do not respond to the same robots.txt rules.
This is a technical rundown of the crawlers actually documented by OpenAI, Anthropic, Google, Apple, and Perplexity: what each one does, and how to see and control what is hitting your server. If you are looking for guidance on writing content that AI models prefer to cite, we have covered that separately in our guide to structured data for LLMs. This post is about access, not content.
Before getting into specific bots, it helps to sort them into two buckets. Mixing these up is the most common mistake we see when site owners write robots.txt rules for AI traffic.
Training crawlers collect pages for inclusion in a future model's training data. They run on their own schedule, have nothing to do with any specific user request, and the content they gather may show up, in some transformed way, in a model months later.
Retrieval crawlers (sometimes called "user" or "search" agents) fetch a page in response to something happening right now. A user asked a question, the assistant decided your page was relevant, and it needs the current content to answer or to cite you. This is the traffic that actually produces the kind of citation Bold GEO tracks across ChatGPT, Perplexity, Gemini, Claude, and Copilot.
Put names to it. GPTBot, ClaudeBot, Google-Extended, and Applebot-Extended are training crawlers. OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, and Perplexity-User are retrieval crawlers. Keep that split in mind through the rest of this list, because it changes what "blocking" actually accomplishes for each one.
Blocking one category does not block the other. That distinction drives almost every decision in this post.
One robots.txt quirk worth flagging before you write a single rule: a group under the wildcard User-agent: * is not guaranteed to apply to any crawler below. Several of these companies document that their crawler looks for a rule group matching its own name first, and only falls back to the wildcard group if no dedicated one exists. Write a named group for every crawler you actually want to control, rather than assuming a catch-all rule covers them.
These are the agents whose stated job is gathering data for model training, documented directly by the companies that run them.
OpenAI's crawler documentation describes GPTBot as the crawler used to make its generative AI foundation models more useful and safe. Disallowing it in robots.txt tells OpenAI your content should not be used to train future foundation models.
User-agent: GPTBot
Disallow: /
Anthropic's crawler documentation describes ClaudeBot as the crawler that collects public web content to help improve its generative models. The same help center article notes support for the non-standard Crawl-delay directive, so you can slow it down instead of blocking it outright.
User-agent: ClaudeBot
Crawl-delay: 1
Older Anthropic user agents, Claude-Web and anthropic-ai, are deprecated. If an old robots.txt file still has rules for them, those rules are not doing anything anymore.
This one trips people up because it is not a crawler at all. Google's own developer documentation is explicit that Google-Extended is a control token, not a separate fetcher: the actual crawling is still done by Googlebot. Disallowing Google-Extended opts your already-crawled content out of training for Gemini and Vertex AI, without touching Search indexing, ranking, or AI Overviews.
Apple runs a similar pattern. Apple's support documentation on Applebot describes Applebot-Extended as a signal that governs how content already fetched by the regular Applebot may be used to train Apple's foundation models, including the models behind Apple Intelligence. Disallowing Applebot-Extended does not stop Applebot from crawling for Siri, Spotlight, or Safari features. It only opts you out of the training use.
These are the ones that matter most if what you care about is whether your brand appears in AI generated answers.
OpenAI documents OAI-SearchBot as the crawler that surfaces websites in ChatGPT's search results, separate from GPTBot's training role. ChatGPT-User is different again: it fires when someone is actively chatting and the assistant needs to open a specific page related to what the user asked. OpenAI treats these as independently controllable, so you can allow search inclusion while still blocking training:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
Anthropic runs a similar three way split to OpenAI's. Claude-SearchBot indexes content to improve search result quality, and Claude-User fetches a page when a person is actively asking Claude a question that requires current, live content. Both respect robots.txt independently of ClaudeBot.
Perplexity documents two agents with a real behavioral difference worth knowing. PerplexityBot crawls to build the index behind Perplexity's answers, and Perplexity's help center states it will not index the full or partial text of pages disallowed via robots.txt, though the domain and a brief summary may still surface. Perplexity-User is the live, on demand fetch that happens when a user's question requires reading a specific page in real time.
Site owners and researchers have raised disputes over whether Perplexity's on demand fetching always respects disallow rules the way a conventional crawler does. Rather than repeat unverified specifics here, the practical takeaway applies across every provider: check your own server logs rather than assuming a robots.txt line is doing all the work.
Microsoft's Bingbot is the crawler behind Bing search, and its index also powers Copilot's web grounded answers. Microsoft documents BingPreview as a related agent that generates page snapshots. There is no separately named "Copilot crawler" identifying itself in your logs. Copilot's web answers draw on the same Bingbot built index, so a rule aimed at Bingbot is the one that matters here.
Not necessarily, and this is the part most robots.txt guides skip. A few things worth knowing before you start disallowing agents:
Consider a concrete case. A site disallows Google-Extended to opt out of Gemini and Vertex AI training, but leaves Googlebot itself fully open. That site can still rank in Google Search, appear in AI Overviews, and get crawled normally, because Google's documentation treats the ranking crawler and the training opt-out as two separate systems entirely. The same logic applies wherever a provider splits a single brand name across multiple user agents with different jobs.
If your goal is visibility, the retrieval crawlers are generally the ones you want open. If your concern is training data, that is a separate, narrower decision, and you can make it per provider without affecting whether you show up in answers today.
robots.txt is a request, not an enforcement mechanism. It tells well behaved crawlers what you would prefer; it does not physically stop a request from arriving. The only way to know what is actually hitting your server is to look.
Most web servers log a user agent string with every request. If you run Apache or Nginx, the combined log format already includes the user agent as the last field on every line, so a simple search across access.log for each crawler name above, GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, and the rest, will get you real request volume, timing, and which paths are being fetched within a few minutes.
Several of these companies also publish IP address ranges you can cross reference to confirm a request claiming to be, say, GPTBot is actually coming from OpenAI's infrastructure rather than someone spoofing the user agent string, which does happen. If your site sits behind a CDN or reverse proxy, also check whether it is capturing and forwarding the original user agent before you conclude a crawler never showed up. A caching layer can serve a crawler's request from cache without ever logging it at your origin, which will make an active crawler look invisible in logs that only cover the origin server.
This log review is also the fastest way to catch the gap between what you configured in robots.txt and what is actually happening, especially for the retrieval agents where compliance is not always as clean as the documentation implies.
It is easy to conflate these because they are both plain text files that sit in your site root, but they do different jobs. robots.txt is an access control signal: it tells crawlers what they may or may not fetch. llms.txt is a discovery and context aid: it points an AI system toward the pages that best represent your product, so a model that does have access can find your most useful content faster.
Neither replaces the other. A well configured robots.txt controls who gets in the door; a well configured llms.txt tells whoever gets in the door where the good stuff is. We wrote a full walkthrough of setting one up in our llms.txt implementation guide, including which sections to prioritize and how to keep it in sync with the rest of your site.
If you are setting this up for the first time, or auditing an existing setup, this is the order that makes sense:
User-agent: * block to cover AI crawlers; several of them are documented as requiring their own named entry rather than inheriting from the wildcard group.That last step is the one most guides leave out entirely. You can get every robots.txt directive technically correct and still have no idea whether it translated into an actual citation on ChatGPT, Perplexity, Gemini, Claude, or Copilot. That is the gap Bold GEO is built to close, watching how your brand actually shows up across all five with a daily refresh instead of leaving it to guesswork.
Bold GEO monitors how your brand is cited across ChatGPT, Perplexity, Gemini, Claude, and Copilot on a daily refresh. 7-day free trial, no credit card.