GPTBot: what it is, its user agent, and how to block it in robots.txt.

1. What GPTBot is, and who runs it.

GPTBot is the web crawler run by OpenAI, the company that makes ChatGPT. Its crawler documentation says it is used to crawl content that may be used in training OpenAI's generative AI foundation models, and that disallowing it indicates a site's content should not be used for that training. It does not answer anyone in real time, and it does not decide what ChatGPT shows when it searches the web. Section two covers the bots that do.

It has been around since August 2023, when it launched with a two-line opt-out: “User-agent: GPTBot” and “Disallow: /”. The launch documentation said crawled pages were filtered to remove paywalled sources, sources known to aggregate personal data, and text that breaks the company's policies. Every request carries the token GPTBot, it reads robots.txt before it crawls, and it comes from a published list of IP addresses (section three).

It is also one of the busiest crawlers on the web. Cloudflare reported that GPTBot made up 11.7% of AI and search crawler traffic across its network in July 2025, up from 4.7% a year earlier, and 28.1% of traffic from AI crawlers alone. Vercel counted 569 million GPTBot requests across its network in a single month in late 2024. And an analysis of more than 7 billion log events from large sites, published in April 2026 by Chris Long of Nectiv with Botify, found GPTBot activity up about 2.9 times after August 2025.

That growth is why the robots.txt question keeps coming up. But the same study found OAI-SearchBot, the search crawler, growing even faster (about 3.5 times), which is the first hint that GPTBot is the wrong lever for anyone who cares about showing up in ChatGPT's answers.

If you came here for the other side of the name, using ChatGPT in your own content work, that is a different page: our guide to using ChatGPT for SEO work covers the setups and prompts. This page is about the bot that reads your site.

2. GPTBot vs OAI-SearchBot vs ChatGPT-User: four OpenAI bots, four jobs.

OpenAI's crawler page lists four user agents, and it says plainly that each setting is independent of the others. Here is what each one does, how it treats robots.txt, and what a block costs you, as of October 15, 2026:

The split matters because each bot pays you back differently. GPTBot feeds what the model knows by itself, months later. OAI-SearchBot decides whether your pages are in the pool ChatGPT search draws from, and it is the one the crawler page recommends allowing if you want to appear in search results. ChatGPT-User shows up when a person asks ChatGPT about your page, which is the closest of the four to a real visitor. Which crawl feeds which kind of answer is the practical core of LLM SEO.

Three details catch people out. First, ChatGPT-User is not used to decide what appears in search, and because a person triggers each visit, the crawler page says robots.txt rules may not apply to it. Second, a site that opts out of OAI-SearchBot is kept out of ChatGPT search answers but can still appear as a navigational link. Third, when a site allows both GPTBot and OAI-SearchBot, the two can share one crawl to avoid fetching the same page twice, so fewer GPTBot hits in your logs is not proof of a block.

OAI-AdsBot is the newest of the four. It only visits landing pages submitted as ads on ChatGPT, and its data is not used for training. ChatGPT's agent mode is not on the list at all: it identifies itself by signing its requests (a Signature-Agent header pointing at chatgpt.com) rather than with a token you can name in robots.txt.

3. The GPTBot user agent, and how to verify a real visit.

These are the strings the crawler page shows on October 15, 2026. GPTBot is first; note that OAI-SearchBot uses a full Chrome-style prefix while the other three do not:

# GPTBot
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot

# OAI-SearchBot
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot

# ChatGPT-User
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot

# OAI-AdsBot
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-AdsBot/1.0; +https://openai.com/adsbot

Match on the token, not the full string. The version number moves (launch-era copies of the GPTBot string read 1.0, today it is 1.4), and when GPTBot or OAI-SearchBot fetches robots.txt it may add an extra “robots.txt” marker to the string, so that log lines without paths still show which requests were for the rules file. robots.txt itself only ever uses the token (“User-agent: GPTBot”).

Then verify. Anyone can put GPTBot in a user agent header, and scrapers do, because some sites wave known crawlers through. Each bot crawls from its own published IP list. The GPTBot file at openai.com/gptbot.json, checked for this guide on October 15, 2026, was generated on September 22, 2026 and lists 18 IPv4 ranges. The OAI-SearchBot list (searchbot.json) has 39 and the ChatGPT-User list (chatgpt-user.json) has 234, and that last one changes often. A request that says GPTBot from an address outside its list is not GPTBot, and blocking it costs you nothing.

curl -s https://openai.com/gptbot.json | jq -r '.prefixes[].ipv4Prefix'

To see which of the four bots visit and from where, pull the source IPs out of your access log. In the common combined log format the client IP is the first field:

grep -E "GPTBot|OAI-SearchBot|ChatGPT-User|OAI-AdsBot" access.log \
  | awk '{print $1}' | sort | uniq -c | sort -rn | head -20

Use the IP lists to catch fakes, not as your training opt-out: robots.txt is the documented signal, and an IP block also stops the bot from reading your rules. Log checks like this one belong with the crawl-access checks in a technical SEO audit, where you already confirm Googlebot can reach what you want ranked.

4. How to block GPTBot in robots.txt (and how to allow it).

robots.txt lives at the root of each host, for example example.com/robots.txt. blog.example.com reads its own file, not the one on example.com, so every subdomain needs its own copy. The four rules most sites need come first, then the trap that breaks them.

User-agent: GPTBot
Disallow: /

User-agent: GPTBot
User-agent: OAI-SearchBot
Disallow: /

User-agent: GPTBot
Disallow: /members/
Disallow: /courses/

User-agent: *
Disallow: /

User-agent: OAI-SearchBot
Allow: /

Several User-agent lines above one set of rules give all of those bots the same rules, which is how rule 2 covers two bots at once. Rule 4 shows why OAI-SearchBot needs its own group on a locked-down site: without it, the wildcard block catches it too. Changes to the OAI-SearchBot group take about 24 hours to show in ChatGPT search.

Two lines you will see in templates and should not rely on. Crawl-delay: the crawler page does not mention it for any of the four bots, so if server load is the problem, rate-limit at your server or CDN instead. And a ChatGPT-User group: robots.txt may not apply to user-initiated visits, so if you truly need them stopped, block the chatgpt-user.json ranges at the edge, knowing that people asking ChatGPT about your page will then get nothing.

One rule trips up more sites than any other. Under the robots.txt standard (RFC 9309), a crawler follows the group that names it and ignores the wildcard group entirely. So the moment you add a GPTBot group, everything in your “User-agent: *” group stops applying to GPTBot:

User-agent: *
Disallow: /admin/

User-agent: GPTBot
Disallow: /drafts/
# GPTBot now ignores the * group above,
# so /admin/ is open to it unless repeated here.

The fix is to repeat every private path inside each named group. It is dull, and it is the line people forget. robots.txt is one item on a much longer list; the rest of the crawl and indexing basics are in our SEO checklist.

5. Should you block GPTBot? Training vs ChatGPT visibility.

Start with what blocking does not do. It does not touch Google rankings: Google Search crawls with Googlebot, and GPTBot plays no part in it. And it does not remove you from ChatGPT search, because OAI-SearchBot does that crawling under its own rules. A GPTBot-only block leaves your ChatGPT search visibility exactly where it was.

What it does is keep your future pages out of training. That cost shows up slowly. When someone asks ChatGPT a question and it answers without searching, the answer comes from what the model learned in training. Launch a product, change your pricing or publish your best guide after the block, and the model's own knowledge of you stops at the block. Only a search can fill the gap. How often your buyers ask without search is the question to answer first; the AI SEO statistics we track are a starting point.

The case for blocking is the exchange rate. Cloudflare tracks how many pages each AI company crawls for every visit it sends back. For the company behind ChatGPT it measured about 1,217 crawls per referral in January 2025 and about 1,091 in July 2025, against about 38,066 for Claude's maker and about 195 for Perplexity that month. So every visitor ChatGPT sends you costs about a thousand page fetches, and most of that crawling is training: 79% of all AI bot crawling in July 2025, by Cloudflare's count.

Most sites still let GPTBot in. Known Agents, which scans the robots.txt files of the top 1,000 websites every day, reported on October 14, 2026 that 21% of them block it. How to choose:

• Block GPTBot, allow OAI-SearchBot. Publishers and anyone whose content is the product (paid archives, licensed data, course material) who wants it out of training but still found when someone asks ChatGPT to search.
• Allow both. SaaS, ecommerce, agencies and most brands. You want the model to know what you sell, and training is the only route into answers given without a search.
• Block both. Sites that want no reuse by ChatGPT at all and accept zero presence in its answers, bar the odd navigational link.
• Rate-limit instead of blocking. If the problem is server load rather than reuse, throttle at the server or CDN. Crawl-delay is not a documented option here.

If ChatGPT answers matter to you, letting OAI-SearchBot in is the start, not the strategy. The pages still have to be the ones worth citing, which is what ranking in ChatGPT comes down to, and the direct-answer format behind it is covered in answer engine optimization.

6. GPTBot vs ClaudeBot and the other AI crawlers.

The other big assistants split their bots the same way: one for training, one for a search index, one for fetches a user asks for. The tokens differ, and each one needs its own robots.txt line. The ones that matter most for an AI-visibility decision:

ClaudeBot is the closest match to GPTBot: a training crawler with a search bot and a user fetcher beside it, and blocking one does nothing to the others. One difference is worth knowing. Claude's three bots all honor robots.txt and the Crawl-delay directive, while robots.txt may not apply to ChatGPT-User at all. The exact rules per bot are in our ClaudeBot guide.

Two more rows behave differently from the rest. Google-Extended is not a crawler at all: Googlebot does the fetching, and the token only tells Google whether that content may be used for Gemini. Google says it does not affect inclusion or ranking in Google Search, and AI Overviews draw on Googlebot's crawl either way (more in our guide to ranking in Google's AI Mode). And Perplexity documents that Perplexity-User generally ignores robots.txt, for the same reason ChatGPT-User may: a person asked for the page.

The practical rule: decide per job, not per company. A common pattern is to block the training crawlers (GPTBot, ClaudeBot and CCBot) and leave every search crawler open. Some sites also publish an llms.txt file alongside robots.txt; it is a reading list, not an access rule, and no bot treats it as one. Which assistants your buyers actually use is covered in our roundup of AI search engines.

7. How GPTBot gets blocked without anyone deciding to.

Plenty of sites block the ChatGPT bots by accident, and the most common cause is not robots.txt. Since July 1, 2025, Cloudflare has blocked known AI crawlers by default on newly onboarded domains. On September 15, 2026 it moved to three categories (Search, Agent and Training): the new defaults block Training and Agent crawlers on pages that show ads and leave Search crawlers allowed. GPTBot counts as Training and ChatGPT-User as Agent. A block at the edge means the bot never reaches your robots.txt, so check the CDN dashboard before you edit the file.

The other usual suspects: a security plugin or firewall rule that blocks by user agent, a “Disallow: /” left over from a staging site, a wildcard group that blocks everything with no OAI-SearchBot group to override it, and an IP blocklist that happens to catch the published ranges.

The last one is not a block at all. Vercel's December 2024 analysis found that none of the major AI crawlers render JavaScript, GPTBot included, and that 34.82% of ChatGPT's crawler requests hit 404 pages. If your content only appears after client-side JavaScript runs, every OpenAI bot can be allowed and still read an empty shell. Server-render the pages you want read. A five-minute check:

1. Open /robots.txt on every host and find each group that names an OpenAI bot, plus the wildcard group.
2. Check the AI crawler or bot settings in your CDN, host and security plugin.
3. Search a week of access logs for the four tokens (section three).
4. Check the source IPs against the published lists.
5. View the source of a key page with JavaScript turned off and confirm the text is there.

For the Googlebot side of the same checks, run a free SEO audit on the domain. ChatGPT is one surface among many now, and the full map of where people search is in search everywhere optimization.

A worked example: one robots.txt, fixed.

Take a fictional online course company, Brindle Academy, that sells bookkeeping courses. It has a marketing site and free guides on www, a blog subdomain, paid lessons under /lessons/ and a logged-in app under /app/. The company and its numbers are invented; the rules and how the bots treat them are real. The goal was simple: keep the course material out of AI training while staying visible when someone asks ChatGPT for the best bookkeeping course. Someone pasted a “block AI” template:

User-agent: *
Disallow: /app/

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
Disallow: /

It blocks training, and it also blocks OAI-SearchBot, the opposite of the goal. Within a day of the change, Brindle's guides stopped appearing in ChatGPT search answers, leaving only the occasional navigational link. Two weeks of logs on www showed OAI-SearchBot requesting robots.txt (with the robots.txt marker) and nothing else, while ChatGPT-User still fetched 19 pages, because the ChatGPT-User line was never a reliable block. The fixed file:

User-agent: *
Disallow: /app/

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Disallow: /app/
Disallow: /lessons/

In the 14 days after the change, OAI-SearchBot fetched 187 pages on www and 64 on the blog, and GPTBot fetched robots.txt and nothing else. Every request checked out against the published IP lists. The same check caught a scraper sending the GPTBot name from a residential IP range, which the team blocked at the firewall without touching robots.txt.

The rules took an afternoon. The harder finding was what OAI-SearchBot had to work with: nine of the free guides were thin pages written years earlier, and two still quoted last year's course prices. Being crawlable is not the same as being cited. To see whether the fix turns into mentions, one of the AI visibility tools can track how often ChatGPT names you over the following months.

Common mistakes when blocking or allowing GPTBot.

1. Blocking OAI-SearchBot to stop training. Training is GPTBot's job alone. Adding OAI-SearchBot to the block removes you from ChatGPT search answers and protects nothing extra.
2. Blocking GPTBot and expecting to leave ChatGPT search. The reverse mistake. Search runs on OAI-SearchBot, so a GPTBot block leaves your pages in search answers.
3. Trusting a ChatGPT-User line. The crawler page says robots.txt rules may not apply to user-initiated visits. If those visits must stop, block the published ChatGPT-User ranges at the server or CDN.
4. Assuming the wildcard rules still apply. A crawler with its own group ignores the “User-agent: *” group. Repeat every private path inside the GPTBot and OAI-SearchBot groups.
5. Fixing one host. robots.txt is per host. A rule on www does nothing for blog, docs or shop subdomains, which each need their own copy.
6. Expecting a block to work instantly, or backwards. OAI-SearchBot changes take about 24 hours to reach search. A GPTBot block is a signal about your content from here on; the crawler page does not say anything already collected is removed.