Skip to content
Primores AI wiki

GPTBot — Definition

GPTBot

TL;DR: GPTBot is OpenAI’s training crawler — it fetches web content used to train OpenAI’s models. It honors robots.txt, publishes its IP ranges, and is separate from OpenAI’s search and user-fetch bots.

What it means

GPTBot is the user-agent OpenAI uses when it crawls the web to gather content for training its models. It is OpenAI’s training crawler specifically — not the bot that powers ChatGPT search, and not the one that opens a page when a user asks ChatGPT about it. Those are OAI-SearchBot and ChatGPT-User respectively. GPTBot identifies itself with a user-agent token (GPTBot/1.4 as of Oct 2026) and OpenAI publishes the IP ranges it crawls from, so site owners can verify a request really is GPTBot rather than a scraper using its name.

Why it matters

GPTBot is the crawler most site owners decide about first, because training is the use that gives nothing back — no citation, no referral traffic. Blocking it is the canonical “I don’t want to train models for free” move, though fewer sites do it than the headlines suggest: about 8% of the top 10,000 sites block GPTBot in robots.txt, versus about half of news publishers (HasData, Jul–Sep 2026). Infrastructure is starting to decide for site owners: since Sep 15, 2026, Cloudflare blocks training bots by default on ad-carrying pages of newly onboarded domains. Crucially, GPTBot respects robots.txt, so a Disallow line genuinely stops it. Blocking GPTBot does not remove your brand from ChatGPT search — that’s governed by OAI-SearchBot, which you can allow independently. The third OpenAI bot, ChatGPT-User, is different again: OpenAI’s docs say robots.txt rules may not apply to these user-initiated fetches. (One caveat: blocking GPTBot stops future training crawls; content already in past training sets stays.)

How it works / examples

To block GPTBot from your whole site:

User-agent: GPTBot
Disallow: /

To allow ChatGPT search while still blocking training, pair that with User-agent: OAI-SearchBot / Allow: /. Because GPTBot is compliant, robots.txt is enough — you don’t need a firewall rule for OpenAI specifically (you do for non-compliant scrapers like glossary/bytespider).

Sources

By Andrej Ruckij