AI crawlers list: every AI bot, a robots.txt template, and which to block.

1. What an AI crawler is: three jobs, not one.

An AI crawler is a bot that fetches web pages on behalf of an AI company. That definition hides the only thing that matters for a robots.txt decision: what the fetch is for. Every major operator now splits its bots by job, and blocking one job does nothing to the others.

• Training crawlers collect pages that may end up in a model's training data. GPTBot, ClaudeBot, CCBot and Meta-ExternalAgent are the big ones. What an assistant knows without searching comes from this crawl, months after it happens.
• Search crawlers build the index an assistant searches when it answers. OAI-SearchBot does this for ChatGPT search, Claude-SearchBot for Claude and PerplexityBot for Perplexity. They decide whether your page can be cited and linked today.
• User-triggered fetchers visit one page because a person asked an assistant about it: ChatGPT-User, Claude-User, Perplexity-User. They are the closest thing to a real visitor, and several of them do not follow robots.txt.

A fourth kind looks like a crawler and is not. Google-Extended and Applebot-Extended are control tokens: Googlebot and Applebot do the fetching, and the token only tells Google or Apple what they may do with the result. Which crawl feeds which kind of answer is the practical core of LLM SEO, and it is why “block AI” is the wrong unit of decision.

The jobs are also very unequal in volume. Cloudflare reported that training made up nearly 80% of AI bot crawling across its network in July 2025, and it measured how many pages each AI company crawls for every visit it sends back: about 1,091 for the company behind ChatGPT, about 38,066 for Claude's maker and about 195 for Perplexity that month. Search crawlers and fetchers are a small slice of the traffic, and they are the slice that sends people back.

Site owners have noticed the difference. Known Agents, which scans the robots.txt files of the top 1,000 websites every day, found on October 15, 2026 that 21% of them block GPTBot but only 11% block OAI-SearchBot, the crawler behind ChatGPT search. The big sites have mostly learned to split the decision. Plenty of smaller ones still paste a single list that blocks all three jobs at once, which is the mistake the rest of this guide is built to prevent.

2. The AI crawlers list: every major bot by job (October 2026).

These are the AI bots with a documented robots.txt token, sorted by job, as each operator described them on October 16, 2026. The robots.txt column says how the bot treats your rules according to its own documentation. Tokens are case-insensitive in robots.txt, so meta-externalagent and Meta-ExternalAgent are the same line.

A few more tokens you may see in logs, none of which you usually need to name. OAI-AdsBot only visits landing pages submitted as ads on ChatGPT, and Meta-ExternalAds crawls for Meta's advertising products. Google-CloudVertexBot only crawls sites whose owners asked for it while building Vertex AI agents, and Google says it has no effect on Search. Google-GeminiNotebook fetches pages users add as sources (it was Google-NotebookLM until August 2026). ChatGPT's agent mode has no token at all: it signs its requests instead.

The two biggest families have their own deep dives. Our GPTBot guide covers all four OpenAI bots, including why ChatGPT-User is unreliable to block, and the ClaudeBot guide does the same for Claude's three. If Perplexity is where your buyers ask, the crawler is only the first step of Perplexity SEO, and our roundup of AI search engines shows which assistants people actually use for research.

One gap is worth knowing. Not every assistant documents a crawler, and Bytespider, which Known Agents attributes to ByteDance, has no official documentation or IP list to check it against. A table like this one is only as good as the operators' own disclosures, which is why section three is about verifying what actually shows up.

3. AI crawler user agents, and how to verify a real visit.

The token is the part that matters. robots.txt only ever uses the token (“User-agent: GPTBot”), and log filters should match it too, because version numbers move: GPTBot launched as 1.0 in 2023 and reads 1.4 today. These are the full strings for the bots you will see most, as documented on October 16, 2026:

# GPTBot (training)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot

# OAI-SearchBot (ChatGPT search)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot

# ChatGPT-User (user-triggered)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot

# ClaudeBot (training)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)

# PerplexityBot (search)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)

# Perplexity-User (user-triggered)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)

# CCBot (training archive)
CCBot/2.0 (https://commoncrawl.org/faq/)

# DuckAssistBot (search answers)
DuckAssistBot/1.2; (+http://duckduckgo.com/duckassistbot.html)

# Meta-ExternalAgent (training)
meta-externalagent/1.1

A user agent is a claim, not proof. Scrapers send GPTBot or ClaudeBot as their name because some sites wave known crawlers through, and Common Crawl warns that impostors send CCBot's name too. The documented bots each publish the IP ranges they crawl from:

• ChatGPT: openai.com/gptbot.json, searchbot.json, chatgpt-user.json and adsbot.json, one file per bot.
• Claude: claude.com/crawling/bots.json, one file for all three bots.
• Perplexity: perplexity.com/perplexitybot.json and perplexity-user.json.
• Common Crawl: index.commoncrawl.org/ccbot.json, plus reverse DNS that resolves to a hostname ending in crawl.commoncrawl.org.

A request that claims a bot's name from an address outside its list is not that bot, and blocking it costs you nothing. To see who visits in the first place, count a week of requests per token. This works on the common combined log format and lowercases the output so mixed-case tokens group together:

grep -oiE "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|CCBot|Bytespider|Amazonbot|Amzn-(SearchBot|User)|meta-(externalagent|externalfetcher|webindexer)|MistralAI-[A-Za-z]+|DuckAssistBot|Google-Agent" access.log \
  | tr '[:upper:]' '[:lower:]' | sort | uniq -c | sort -rn

Use the IP lists to catch fakes, not as your opt-out. robots.txt is the documented signal for every bot that honors it, and an IP block also stops the bot from reading your rules. Log checks like this one belong next to the crawl checks in a technical SEO audit, where you already confirm Googlebot can reach what you want ranked. For the OpenAI side, the GPTBot user agent walkthrough shows the robots.txt marker those bots add when they fetch the rules file.

4. A robots.txt template for AI bots.

Four rules of the file before the template. robots.txt lives at the root of each host, and blog.example.com reads its own file, not the one on example.com. Several User-agent lines stacked above one set of rules form a single group that shares those rules. A bot that finds a group naming it follows that group and ignores the wildcard group entirely. And Crawl-delay is not universal: ClaudeBot supports it, Applebot and Amazonbot say they do not, and OpenAI's crawler page does not mention it.

This template implements the most common policy: stay visible in AI search and assistant answers, keep future pages out of model training. Swap the private paths for your own, and read section five before you keep group 1 as it is.

# AI crawler rules · reviewed 2026-10-16
# Policy: visible in AI search and assistant answers,
# future pages kept out of model training.

# 1. Training crawlers and training tokens: blocked
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Meta-ExternalAgent
User-agent: Amazonbot
User-agent: MistralAI-Training
User-agent: Bytespider
User-agent: Applebot-Extended
Disallow: /

# 2. AI search crawlers: allowed, minus private paths
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Meta-WebIndexer
User-agent: Amzn-SearchBot
User-agent: MistralAI-Index
User-agent: DuckAssistBot
Disallow: /account/
Disallow: /checkout/

# 3. User-triggered fetchers: same private paths
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
User-agent: Meta-ExternalFetcher
User-agent: Amzn-User
User-agent: MistralAI-User
Disallow: /account/
Disallow: /checkout/

# 4. Optional: Gemini training AND grounding in Gemini Apps.
# Uncomment only if you accept fewer Gemini app answers.
# User-agent: Google-Extended
# Disallow: /

# 5. Everyone else (Googlebot, Bingbot, Applebot...)
User-agent: *
Disallow: /account/
Disallow: /checkout/

What each group does. Group 1 sends the training opt-out to every documented training crawler, plus Applebot-Extended, which only governs training. Groups 2 and 3 keep the search crawlers and fetchers allowed, and they repeat the private paths because those bots no longer read the wildcard group once they are named. Group 4 is commented out on purpose: Google-Extended is not a pure training switch (section five). Group 5 is the existing rule set for Googlebot, Bingbot and everything else.

That repetition is the line people forget, and it is the most common way a “block AI” edit opens a private path to the bots it names:

User-agent: *
Disallow: /account/

User-agent: PerplexityBot
Disallow: /drafts/
# PerplexityBot now ignores the * group above,
# so /account/ is open to it unless repeated here.

Expect a delay after you publish the file. OpenAI says ChatGPT search takes about 24 hours to reflect an OAI-SearchBot change, Perplexity says up to 24 hours, Meta says its crawlers may cache robots.txt for up to a day, and DuckDuckGo quotes 72 hours. robots.txt is one item on a much longer list; the rest of the crawl and indexing basics are in our SEO checklist.

5. Which AI crawlers to block: a decision framework.

Start with what blocking does not touch. Google Search crawls with Googlebot, and no AI crawler plays a part in Google rankings. Google says Google-Extended does not affect inclusion or ranking either, and its AI features documentation says AI Overviews and AI Mode only need a page to be indexed and eligible for a snippet. Every decision below is about AI assistants, not Google rankings.

Then decide per job:

The training decision is the hard one, because the cost of each side shows up slowly. Allow training and your pages shape what models say about you when they answer without searching, with no click and no credit in return. Block it and that knowledge freezes at the block: a product launched, a price changed or a guide published afterwards only reaches answers through a search. How often your buyers ask without search is the question to settle first, and the AI SEO statistics we track are a starting point.

What the biggest sites chose, from the Known Agents scan of the top 1,000 websites on October 15, 2026. Training crawlers are blocked about twice as often as the search crawler behind ChatGPT:

By type of site:

• SaaS, ecommerce, agencies and local services: allow everything documented. You want models to know what you sell, and training is the only route into answers given without a search. Block only bots you cannot verify.
• Publishers, paid archives, courses and data businesses: block training, allow search. The template in section four, as written. Your content is the product, and the search crawlers still send readers back.
• Sites that want no AI reuse at all: block every named group. Then block the user-triggered fetchers at the edge, because robots.txt will not stop all of them, and accept that assistants will have nothing of yours to cite.
• Sites struggling with load: rate-limit instead of blocking. Throttle at the server or CDN. Crawl-delay only works for the bots that support it.

Two fine-print calls. Applebot-Extended is a clean training opt-out: Apple says pages that disallow it can still appear in its search results. Google-Extended is not. Google's documentation says it also governs grounding in Gemini Apps, so blocking it can reduce how often the Gemini app answers from your pages. That is why the template leaves it commented out.

Letting the search crawlers in is the start, not the strategy. The pages they find still have to be the ones worth citing, which is the subject of generative engine optimization, and for ChatGPT specifically, of ranking in ChatGPT.

6. What robots.txt cannot do: fetchers, tokens and the CDN.

robots.txt is a request that well-behaved bots agree to honor. Four things sit outside it, and each one explains a “but I blocked it” or “but I allowed it” that shows up in the logs.

User-triggered fetchers. Perplexity says Perplexity-User generally ignores robots.txt, Google says the same of its user-triggered fetchers including Google-Agent, Meta says Meta-ExternalFetcher may bypass it, OpenAI says it may not apply to ChatGPT-User, and Amazon says Amzn-User may not follow every directive. The reasoning is the same everywhere: a person asked for the page. If those visits must stop, block the published IP ranges at the edge, knowing that people asking about your page will then get nothing.

Control tokens. A Google-Extended or Applebot-Extended rule does not stop a single request. Googlebot and Applebot keep crawling as before, and the rule only changes what the data may be used for. If you expected fewer hits after adding them, that is why.

The CDN layer. Since July 1, 2025, Cloudflare has blocked known AI crawlers by default on newly onboarded domains. Its changelog describes three categories you can now control separately (Search, Agent and Training), and from September 15, 2026 new domains get Training and Agent bots blocked on pages that display ads, with Search allowed. The same entry warns that multi-purpose crawlers that combine search and training are affected by the Training default, so check what it does to Googlebot and Applebot before you accept it. A block at the edge means the bot never reaches your robots.txt.

Rendering. Vercel's December 2024 analysis found that none of the major AI crawlers render JavaScript. A page whose text only appears after client-side scripts run can be fully allowed and still read as an empty shell. Server-render the pages you want cited.

And one file that is not an access rule at all. An llms.txt file is a reading list for language models, and no crawler treats it as permission or a block. Google says you do not need new machine-readable files or AI text files to appear in its AI features; the controls there are Googlebot and the snippet directives (nosnippet, data-nosnippet, max-snippet), covered in our guide to ranking in AI Overviews. A five-minute check that covers all four gaps:

1. Open /robots.txt on every host and list each group that names an AI bot, plus the wildcard group.
2. Check the AI bot settings in your CDN, host and security plugin, including any custom firewall rule that matches user agents.
3. Count a week of requests per token (section three).
4. Check the source IPs of the top tokens against the published lists.
5. View the source of a key page with JavaScript turned off and confirm the text is there.

AI assistants are one surface among many now. The full map of where people search, and what each surface needs from a page, is in search everywhere optimization.

A worked example: one SaaS site, one week of logs.

Take a fictional invoicing app for freelancers, Ledgerline, with a marketing site and help center on ledgerline.io, a logged-in app under /app/ and an API under /api/. The company and its numbers are invented; the bots and how they behave are real. Ledgerline sells software, so the goal was the SaaS policy from section five: be known to the models, be citable in AI search, keep the app private. Before touching robots.txt, the team counted a week of requests per token:

requests · 7 days · ledgerline.io (invented numbers)

bytespider             9,840   0 from a verifiable list
gptbot                 3,110
claudebot              2,460
meta-externalagent     1,920
ccbot                  1,140
amazonbot                880
oai-searchbot            640
chatgpt-user             212
claude-user               38
perplexitybot              0
perplexity-user            0

Two lines stood out. Bytespider was the busiest AI bot on the site, and none of its requests could be checked, because there is no published list to check them against. And both Perplexity bots were at zero, on a site whose buyers use Perplexity for software research. The robots.txt file did not mention Perplexity at all. The cause was a custom firewall rule from 2024 that blocked any user agent containing “Perplexity”, written by a contractor after a news story and forgotten.

# ledgerline.io/robots.txt · reviewed 2026-10-16
User-agent: *
Disallow: /app/
Disallow: /api/

# No documentation, no IP list, 9,840 requests a week.
# Also rate-limited at the CDN, because this line is a request.
User-agent: Bytespider
Disallow: /

The whole file is seven lines of rules, because the right policy for a SaaS site is mostly “allow”, and the wildcard group already does that work. The fix that mattered most was not in robots.txt at all. In the 14 days after the firewall rule came out, PerplexityBot fetched 412 pages and Perplexity-User 57, and the help center subdomain, which serves its own copy of the file, showed the same pattern.

The harder finding came from reading what the fetchers fetched. Of the 212 ChatGPT-User requests in the first week, 61 went to three old comparison pages that still quoted last year's plans. The crawlers could reach Ledgerline fine; what they found was out of date. To see whether the access fix turns into mentions over the following months, one of the AI visibility tools can track how often each assistant names you.

Common mistakes when blocking AI crawlers.

1. Pasting a “block all AI bots” list without reading it. Most of those lists mix training crawlers with OAI-SearchBot, Claude-SearchBot and PerplexityBot, so the site drops out of AI search answers while trying to opt out of training. Sort the list by job before you paste anything.
2. Blocking a training crawler and expecting to leave AI search. A GPTBot or ClaudeBot block does nothing to ChatGPT search or Claude's search results, which run on their own search crawlers. The reverse is also true: blocking a search crawler does not opt you out of training.
3. Forgetting the wildcard trap. A bot with its own group ignores the “User-agent: *” group. Repeat every private path inside each named group, or naming a bot opens paths it could not reach before.
4. Trusting robots.txt to stop user-triggered fetchers. Perplexity-User, Google-Agent and Meta-ExternalFetcher document that they may skip it, and so do ChatGPT-User and Amzn-User. If those visits must stop, block the published IP ranges at the edge.
5. Treating Google-Extended as an AI Overviews switch. It is not a crawler and does not touch Google Search or AI Overviews, which run on Googlebot. What it does cover, Gemini training and grounding in Gemini Apps, is exactly what a visibility-minded site wants to keep.
6. Fixing robots.txt once, on one host, and never again. Each subdomain reads its own file, the CDN or firewall can override all of it, and the token list keeps moving: OAI-AdsBot and Google-Agent are recent additions, and Google-NotebookLM became Google-GeminiNotebook this year. Date the file and review it every quarter.