Nobody Voted For The Company That Now Licenses The Web

Cloudflare rolled out a new pricing structure on July 1, swapping its long-standing “pay-per-crawl” model for a “pay-per-use” scheme that pays publishers each time an AI system consumes their content. The change also introduced automatic blocking of mixed-use crawlers on pages that contain ads, effective September 15. The shift turns the CDN and security provider into the de-facto gatekeeper of who may read the web and on what terms, a move that could reshape the economics of AI training and the broader internet ecosystem.

Why the pricing tweak matters

Under the old system, a bot that wanted to fetch a page paid Cloudflare a fee per request. The fee went to Cloudflare, not to the site owner. The new model flips that flow: when an AI model accesses a page, the publisher receives a payment from Cloudflare. The payment part is good. It gives money to site owners.

The practical effect hits anyone building an AI crawler right away. If the crawler is classified as “mixed use” – meaning it both indexes pages for search and extracts content for training – Cloudflare blocks it by default on any page that serves advertisements. To get access, the operator must either pay Cloudflare’s rate or accept that those pages are off-limits.

The policy scaffolding behind the switch

Cloudflare’s product specification groups AI crawlers into three buckets:

  • Search – bots that index pages to power a search engine.
  • Agent – bots that retrieve information for a conversational AI or similar agent.
  • Training – bots that harvest text, images, or other data to train a machine-learning model.

The categorisation isn’t dictated by law or an industry standard; Cloudflare’s internal team decides it. That team also assigns each bot to a bucket, for example whether a research-oriented crawler counts as an “agent” or a “trainer,” or whether an archival bot counts as a “search” bot.

These classifications directly affect whether a bot is blocked, how much it is charged, and whether the publisher receives a payment.

Who wins, who loses

Publishers

Site owners now earn a new revenue stream. Every time an AI model reads their article, Cloudflare sends them a check.

AI developers

Start-ups and research labs that rely on large, diverse datasets now face an extra cost layer. If a crawler isn’t on Cloudflare’s approved list, the developer must either pay the per-use fee or lose access to a significant slice of the web.

The business logic behind scarcity

Cloudflare’s model creates scarcity to run a marketplace. By default, mixed-use bots are blocked on ad-filled pages, turning every request into a billable event. If the company blocked all bots, it would erase its own revenue stream. The selective blocking therefore protects the bottom line and forces bot operators into a paying relationship.

The shift also exploits Cloudflare’s massive free-tier user base. Millions of sites signed up because the free plan offered easy CDN and security services. That “free” access was a customer-acquisition strategy, not a charitable offering. Now those sites belong to a network that generates revenue for Cloudflare when AI consumes their content, but the sites have no say in the pricing or the policy categories that determine who gets paid.

Takeaway

Cloudflare’s pay-per-use overhaul does more than add a new revenue line; it gives the company the power to label, charge, and ultimately decide which AI crawlers can read the web.