Cloudflare Blocks AI Crawlers on New Sites by Default

In July 2025, Cloudflare began blocking AI crawlers by default on new sites. A decade earlier, CEO Matthew Prince estimated Google crawled two pages for every visitor it sent a publisher: 2:1. No invoice required.

By 2025, the Cloudflare CEO put Google’s ratio at 18:1 and OpenAI’s at 1,500:1. These are Prince’s comparisons, not an audit of every publisher or a price for any individual page. Google and OpenAI also serve different products. But the ratios expose what the publisher referral bargain depended on: a crawler returning an audience.

Cloudflare is moving machine access from that exchange into publisher-directed rules at the delivery layer: identify a requester, allow or deny it, and potentially charge it. Cloudflare serves a reported 20% of the web, making its default consequential but far from universal.

Cloudflare shifted the block from opt-in to default

Cloudflare had long controlled automated traffic. Its opt-in Bot Fight Mode challenged suspected bots in 2019. In July 2024, it gave all customers a free, one-click AI-bot block, but site owners still had to turn it on.

A year later, Cloudflare introduced Pay per Crawl and made blocking AI crawlers the starting setting for new sites. It did not announce the same automatic change for existing sites.

GoDaddy subsequently put Cloudflare AI Crawl Control into its hosting platform, where site owners can block, permit or potentially monetize crawler access. Beehiiv added the controls to writers’ dashboards. The integration reports give no count of participating sites or writers, or a breakdown of who chose block over allow.

A priced crawl needs an identifiable buyer

Cloudflare alleged in 2025 that Perplexity used undeclared user agents and rotating IP addresses to get around robots.txt instructions and network blocks. The allegation illustrates the enforcement problem without establishing how often evasion succeeds: a site cannot apply different terms to search, training and agents if it cannot reliably tell which requester has arrived.

Cloudflare set a September 15, 2026 deadline for AI companies to separate search, AI-training and AI-agent crawlers or face blocking. A shared traffic-classification protocol developed with Google, Microsoft and Mozilla addresses the related question of which requests are legitimate. Both efforts matter to a publisher setting a price: it needs to know who requests a page and how the requester intends to use it.

Cloudflare said it had blocked 416 billion AI-bot requests for customers since July 1, 2025. That counts enforcement actions, not licensed crawls or foregone publisher revenue. A block applies to traffic Cloudflare can recognize; it cannot guarantee that every machine observes a publisher’s terms.

Permission is spreading faster than settlement

Cloudflare described Pay per Crawl as a private beta with a small number of publishers and creators. The cited reports name no completed per-crawl purchase with a public price, named buyer, settlement mechanism or publisher payout. A marketplace needs transactions, not just a place for sellers to state terms.

Other companies are building different routes to permission. OpenAI, Google, Microsoft and Anthropic said they would adopt a shared publisher opt-out standard across crawlers. A portable preference could reduce publishers’ dependence on any one CDN’s policy interface; it would not itself collect payment or stop an unidentified bot. Microsoft’s Publisher Content Marketplace, launched with Condé Nast, Hearst, AP and others, instead offers terms, usage tracking and payment for licensed content used to ground AI answers. That is a different transaction from charging for an HTTP request.

The old 2:1 required no invoice because the reader was the return. Cloudflare can count blocked requests, but no public record yet shows a publisher paid for a crawl.