AI crawler user agents

The AI crawler user agents, and what each one is for

There are 11 AI crawler user agents and robots.txt control tokens worth knowing about, from 5 vendors, and they do three different jobs. 5 of them build a search index — block one and the assistant behind it can no longer find your pages. 3 fetch a single page while a user is mid-question. 2 collect training data, and blocking those does not remove you from anybody’s search results. One entry in the list is not a crawler at all. Every row below was checked against the vendor’s own bot documentation on 15 August 2026, and the doc link in each row is the page it was checked against.

Vendors: Anthropic, Google, Microsoft, OpenAI, Perplexity. Rendered from the probe table the audit tool itself uses, so this page cannot drift from the tool.

Blocking the wrong one costs you the answer, not the training set

The distinction that most robots.txt advice misses: an assistant’s search crawler and its training crawler are different tokens with different consequences.

• Search-index crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot, bingbot). These decide whether the assistant can retrieve your page while answering. Disallow one and you are not in that assistant’s answers, no matter how good the page is.
• On-demand fetchers (ChatGPT-User, Perplexity-User, Claude-User). These arrive because a user asked something that needs your page right now. Blocking them breaks the live look-up but leaves the index alone.
• Training crawlers (GPTBot, ClaudeBot). These feed model training. Many sites block them deliberately, and that is a legitimate choice with no cost to search visibility.

The gptbot robots.txt rule almost everyone gets backwards

GPTBot is OpenAI’s training crawler. A User-agent: GPTBot / Disallow: / block keeps your pages out of training data and does nothing to your visibility inside ChatGPT’s search — that job belongs to OAI-SearchBot. So a rule meant to opt out of training that disallows both tokens does two things, and only one of them was intended: it opts you out of training, and it takes you out of the answers. Check which tokens your own robots.txt names before you assume it does what you meant.

The same shape repeats across vendors: ClaudeBot is training, Claude-SearchBot is search. Google-Extended is the odd one — Google’s own documentation describes it as a robots.txt control token rather than a crawler, and it publishes no user-agent string at all.

A robots.txt that blocks AI crawlers for training and keeps you findable

Copy this, keep the comments, and change it deliberately rather than by pattern-matching someone’s blog post:

# The crawlers that decide whether an assistant can find you at all
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: bingbot
Allow: /

# Training crawlers — block these if that is your choice
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Two limits worth stating. robots.txt is voluntary — it is a request, not an access control; anything you need actually protected needs authentication. And a Disallow on Google-Extended has no effect on Google Search indexing or ranking, which is a separate system.

The full user-agent strings, and how to test your own site

Some firewalls match on the whole user-agent string rather than the token, so sending only the token can produce a pass that a real crawler would not get. These are the strings the vendors publish:

To see what your own server does with one of them, send the request yourself:

curl -sI -A 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot' https://your-site.com/

A 200 means that crawler can read the page. A 403 — the usual symptom of a WAF or bot-protection rule, not of robots.txt — means it cannot, and no amount of content work will fix that.

Three things this table does not pretend

• Three of these strings are bare tokens. Anthropic documents Claude-SearchBot, Claude-User and ClaudeBot by name without publishing a full browser-style user-agent string, so that is what the table shows — the token, not a reconstruction of it.
• One row is a control token, not a crawler, and the table says so rather than padding the count. Treating Google-Extended as a crawler is the single most common error in published AI-crawler lists.
• We have not verified how ChatGPT’s web search retrieves pages. Our own probe table carries a note that it rides on the Bing index, which is why bingbot is in the scored set — but no vendor document we checked states it, so treat it as a working assumption rather than a fact, and weigh bingbot accordingly.

If you want the same check run against your own site, with the reachability of each of these crawlers recorded as a scored audit line, that is one dimension of the $390 audit. Two of the assistants above are wired up today; three are not, and the home page keeps that list current.

AI crawler questions we get asked

Where these numbers come from

• OpenAI — bots and user agents (OAI-SearchBot, ChatGPT-User, GPTBot)
• Perplexity — Perplexity Crawlers (PerplexityBot, Perplexity-User)
• Anthropic — does Anthropic crawl data from the web (Claude-SearchBot, Claude-User, ClaudeBot)
• Google — common crawlers (Googlebot, Google-Extended). Both this and the Perplexity link moved since the table was verified; the new addresses were re-checked on 5 September 2026.
• Bing — bingbot user-agent change
• Our own probe table, geo_radar/probes.py, which is where the rows above are rendered from — each entry carries the doc URL it was checked against and the date it was checked.

Related

• Perplexity SEO: what is actually verifiable — the two Perplexity tokens, and what we have and have not measured
• How to rank in ChatGPT — what our own run showed about which pages get cited
• The GEO audit rubric — crawler reachability is 6 of the 100 points, and here is the rest