Blocking GPTBot does not erase your site from ChatGPT’s answer pane. Site owners who added a robots.txt rule against the bot were surprised to see their articles still show up with links and excerpts when users ask ChatGPT about the topic. The block worked – it just stopped the wrong crawler.

Training bots vs. citation bots

AI providers run two distinct kinds of crawlers:

  • Training bots scrape publicly available pages, ingest the text, and use it to fine-tune large language models. They never return a link to the source, and they generate no traffic for the site.
  • Citation bots operate at query time. When a user asks a question, the bot searches the web, pulls a relevant snippet, adds the source’s name and URL, and presents it in the chat window. Those hits can drive real visitors.

If you block the training bot, the model can no longer learn from your pages. If you block the citation bot, the page disappears from ChatGPT’s live answers. The confusion arises because many sites block “GPTBot” without realizing that the same name does not appear in the citation workflow.

How the big players split their crawlers

Provider Training-bot name Citation-bot names
OpenAI GPTBot OAI-SearchBot, ChatGPT-User
Anthropic ClaudeBot Claude-SearchBot, Claude-User
Google Google-Extended Googlebot
Perplexity — (both used for citations) PerplexityBot, Perplexity-User

Only the entries under “Training-bot name” need to be denied if you want to keep your content out of future model training. The citation bots should stay allowed if you wish to appear in ChatGPT’s real-time answers.

Configuring robots.txt the right way

A generic Disallow: / line that targets “GPTBot” blocks the training crawler but leaves the citation bots untouched. To be explicit, add rules that deny each training bot while permitting the citation bots:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

# Allow citation bots
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: Googlebot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

Do not rely on a default robots.txt that only blocks the generic “*”. The distinction matters because a blanket block would keep both training and citation bots out, cutting off the traffic that AI-driven search can provide.

Cloudflare users: use the AI Crawl Control panel

If your domain is behind Cloudflare, you may see a “Cloudflare management marker” inside the robots.txt file. Editing the file manually can cause the changes to be overwritten at the next Cloudflare update. Instead, navigate to the Cloudflare AI Crawl Control interface and toggle the switches for the relevant bots. The panel writes the correct directives behind the scenes and keeps them persistent.

Why the nuance matters

  • Intellectual-property protection – Blocking training bots prevents your prose from being harvested for free and incorporated into future models.
  • Referral traffic – Allowing citation bots means users who see a snippet in ChatGPT can click through to your site, generating real visits.
  • Brand visibility – Presence in AI-augmented search keeps your content discoverable as AI assistants become a primary information source for many users.

The counter-argument

Some publishers argue that any use of their text, even for citation, contributes to the broader AI ecosystem and that refusing citation bots could hurt their reach. The trade-off is clear: you either let the model learn from your work without credit, or you let it quote you and drive traffic. The choice depends on how you value immediate clicks versus long-term control over how your words are reused.

What to watch next

AI companies are still iterating on how they name and route their crawlers. Keep an eye on updates to the user-agent strings in their developer documentation. A new citation bot could appear under a different name, and a previously blocked training bot might be renamed. Regularly audit your robots.txt and Cloudflare settings to stay aligned with the latest crawler roster.

Takeaway: Block only the training-bot user agents to keep your content out of future model training, but leave the citation bots enabled so ChatGPT can still point users back to you. The right robots.txt rules, applied through Cloudflare’s AI Crawl Control when needed, let you protect your IP while harvesting AI-driven traffic.