Cloudflare sets a September 15 deadline for AI companies to differentiate their web crawlers into search, AI training, and AI agents or face being blocked
Cloudflare gave AI crawlers a September deadline to separate the bots that gather content for search from those that harvest …
NBC NewsSamantha Elkins
Context & Ripple Effects
Cloudflare’s policy has progressed from free tools for detecting and blocking AI scraping in 2024 to default blocking for new sites and a proposed “Pay per Crawl” market in 2025. The company has also emphasized the scale of AI-bot traffic reaching its customers.
The new requirement turns that broader access-control effort into a bot-identity rule: publishers and site operators can make different decisions about discovery, model training, and agent activity rather than treating all AI crawling alike.
First-order effects
AI companies using Cloudflare-protected sites must identify crawlers by purpose before the deadline or risk losing access to those sites.
Cloudflare customers gain more granular control over whether their content is available to search-oriented bots, training collectors, or AI agents.
Second-order effects
Crawler operators will need to separate their infrastructure and declarations by use case, making it harder to rely on a single broadly authorized bot identity.
The policy strengthens the practical foundation for paid or negotiated access, since sites can distinguish traffic they may tolerate for search from traffic they want to restrict or monetize.
Third-order effects
If widely adopted, purpose-specific bot controls could make web access for AI systems less like an open crawling environment and more like a permissioned, auditable market.
The approach could also create a de facto expectation that AI companies disclose how collected web content will be used; its durability depends on enforcement and whether other web intermediaries follow Cloudflare’s lead.
The trend: AI-data access is shifting from undifferentiated web scraping toward granular controls, attribution, and potential compensation for distinct uses of publisher content.
The new business model that pays for the Internet in a world of AI agents instead of humans browsing. Will take a while as most humans are still using AI as a siloed browser experience, but as soon as they start using AI like everyone in tech, the model will flip and quickly.
This is really cool. I wrote this post 8 years ago. A good reminder that tech adoption take longer to happen than you think it will, and then happens faster than you ever thought it could. [image]
We're opening the waitlist for our Monetization Gateway, which will allow you to charge for any web page, dataset, API, or MCP tool behind Cloudflare. The charges will settle in stablecoins over the x402 open protocol. https://blog.cloudflare.com/ ...
I'm excited to announce that I've joined @Cloudflare as a Product Manager focused on Agent Payments. We are building out payment tooling on top of stablecoin rails, and our first product is a Monetization Gateway that will help website creators get paid for their content and APIs
Thank you once again @eastdakota for standing up for what's right for the Internet. What I'm not clear about is how you plan to deal with user initiated agents.
For our second Content Independence Day, we're giving website owners finer options to manage AI traffic. Find out more: https://blog.cloudflare.com/ ...