Cloudflare’s New AI Crawler Controls
Website traffic is changing.
In the past, search engines crawled your site to help people find you. It was a simple deal. AI has made this much more complex.
A bot might crawl your page to show it in a search result. That is different from a bot crawling your page to train an AI model.
Cloudflare is changing how you manage this. On July 1, 2026, all customers get new AI traffic controls. This allows you to separate traffic into three groups:
- Search: Bots that index content for search engines or AI answers.
- Agent: Bots acting on behalf of a user to complete a task.
- Training: Bots collecting data to train or fine-tune AI models.
This shift turns visibility into an access policy decision. It is no longer just an SEO decision.
If you run a SaaS company, your public pages like documentation, pricing, and help centers create value. They help you acquire and support customers.
You must decide how much access you want to give.
The old question was: Can search engines find us? The new question is: Which automated systems should use our site, and for what purpose?
Do not treat all AI bots as the same.
Search bots help users find your product. Blocking them might make you invisible in AI-powered search results.
AI agents can be useful. An agent might visit your pricing page to help a user compare plans. However, you must decide if these agents need authentication or rate limits.
Training bots offer less direct value. They use your content to build models but may not send users back to your site. Many companies choose to block these.
A smart policy looks like this:
- Allow search bots to index your product and help pages.
- Allow user-directed agents to access public documentation.
- Block or license access for training bots.
- Keep all authenticated product areas protected.
Review your traffic regularly. Look at crawler categories, server load, and search visibility.
The goal is not to block every bot. The goal is to control what they do.