Cloudflare debuts “Pay per Crawl”, a marketplace that lets sites charge AI crawlers per crawl; new sites using Cloudflare will now block AI crawlers by default
It's a total reshaping of how people make money on the internet. Full story: techcrunch.com/2025/07/01/c... Drew Harwell / @drewharwell.com : “Deeply concerned that the incentives for content creation are dead.” — www.nytimes.com/2025/07/01/t... Nilay Patel / @reckless : Big move here - AI firms are all facing down an existential content theft reckoning and still haven't explained what applications will generate enough value to make it worth it www.theverge.com/news/695501/ ... Joe Fabisevich / @mergesort.me : Really hoping for the sake of @plinky.app that this new opt-into scraping default doesn't mean small well-intentioned scrapers like mine don't suddenly stop working. 😬 Chris Brandrick / @brandrick : This sorta thing felt like an inevitability — interesting to see someone actually have a go at it. Be curious to see how it plays out. [embedded post] @iaaa.org : arstechnica.com/tech-policy/ ... In a blog Tuesday, Cloudflare explained that its “pay-per-crawl” feature is currently in a private beta. A small number of publishers and content creators will participate in the experiment. @cloudflare.social : Pay per crawl is a new feature to allow content creators to charge AI crawlers for access to their content. blog.cloudflare.com/introducing- ... Michael Weinberg / @michaelweinberg.org : I really do not like this “a small number of entities get to monetize being scraped on the open internet” approach to AI training data conflicts — arstechnica.com/tech-policy/ ... Mastodon: Richard MacManus / @ricmac@mastodon.social : “Mr. Prince said he was “deeply concerned that the incentives for content creation are dead. This definitely resonates with me. Sometimes I wonder why I bother spending so much of my time writing articles that fewer and fewer people read. … @kaiserkiwi@corteximplant.com : That's nice and stuff, but what when I don't want to route my users through a third party? — It seems it's more and more a weakness to protect the user's data. — But I don't see me locking off my sites just because some people take “the free web” too serious. … @nixCraft@mastodon.social : Cloudflare introduces pay-per-crawl for AI bots https://blog.cloudflare.com/ ... We don't need another technical solution to this madness. We need legislation that blocks all batshit crazy AI companies like OpenAI, Microsoft, Google, and others from stealing from artists & writers … Dare Obasanjo / @carnage4life@mas.to : Cloudflare is launching a new feature for websites it hosts to charge AI chatbots for crawling called Pay per Crawl. This goes along with a change to block AI crawlers by default on new websites. — The current model is unsustainable and I expect the best content on the web to eventually be paywalled for both humans and bots outside of a few large scale ad supported services. … @arch@floofy.tech : Time will tell how this shakes out...... https://blog.cloudflare.com/ ... X: Ed Newton-Rex / @ednewtonrex : Cloudflare will block AI crawlers by default. This is the way. Stands in stark contrast to companies like Squarespace, who opt you in to AI training by default. If Cloudflare adds a drag-and-drop website builder, Squarespace's days are surely numbered. https://www.wired.com/... [image] Matthew Prince / @eastdakota : Proud of the Cloudflare team working to build a sustainable business model for the AI-driven web. Happy Content Independence Day! No AI-scraping without compensation. [video] Nate Hake / @natejhake : I'm SUPER happy Cloudflare is doing this marketplace, and even happier William says they aim for it to be open to small & indie creators. Bravo 👏 A serious concern I have is what Google told the UK in its copyright consultation: “No single piece of content has value, and as such, pricing becomes a pure bargaining issue.”... Artem Zakharchenko / @kettanaito : The problem with this approach is that it implies a one-time transaction whereas most creative content is licensed on a per-usage basis. Crawl !== usage as an agent might crawl your site once and use it in their training data indefinitely. @lukemulks : When Cloudflare decides to do things, they can change a default for 19.5% of the web and frame it as “Content Independence Day.” 🤷♂️ Mic King / @ipullrank : Cloudflare's pay per crawl model is an interesting one, but I don't think we live in a world where it sticks. If anything I think this will accelerate the need for caching by AI bots. I love that people are brainstorming things like this and hope we keep going til we find some Matt / @mattmaximo1 : And they did it without launching a token Incredible. @niemanlab : On Tuesday, Cloudflare became the first major internet infrastructure company to block AI scraping by default. The shift from an “opt-out” model to an “opt-in” model means AI companies need to receive explicit permission from a website serviced by Cloudflare before scraping. Ed Newton-Rex / @ednewtonrex : The Cloudflare AI crawler blocker is great, but it only gives people control over their work on websites they control. It's like having body armor that stops working when you leave your house. Respect to Cloudflare - there's only so much one company can do. Just remember that [image] Ted Nyman / @tnm : a long-standing dream even before LLM crawling. this is great. Aly Cabral / @aly_cabral : LLMs are reshaping how we access information. It's time to rethink how we compensate original content creators in this new world. https://techcrunch.com/... @cloudflare : It's Content Independence Day: Cloudflare, along with a majority of the world's leading publishers and AI companies, is changing the default to block AI crawlers unless they pay creators for their content. https://blog.cloudflare.com/ ... Arvid Kahl / @arvidkahl : Now THIS is exciting for public content platforms. Cloudflare turns into an MoR for per-crawl micropayments. Still looks like quite the challenge to implement, but I do like where this is going. [image] Chris Stokel-Walker / @stokel : I foresee a future where AI model providers pay to crawl sites for their training data which helps parse information fed to LLMs that users pay per million token API access to and the whole cycle repeats, handing money back and forth https://techcrunch.com/... Ilya Grigorik / @igrigorik : This is huge, and much needed. Bot traffic vastly outnumbers human requests. Bots are not bad, but identifying the good ones is really challenging. Dane Knecht / @dok2001 : An open Internet depends on a model that incentivizes great content to continue to be published. Proud to partner with some of the best publishers and AI services to launch a new model built that rewards high-quality material in a new AI era. Aleyda Solis / @aleyda : .@Cloudflare is introducing a pay per crawl system: Enabling content owners to charge AI crawlers for access, offering a new monetization method for access to content 👇 “We believe your choice need not be binary — there should be a third, more nuanced option: You can charge for [image] Meredith Whittaker / @mer__edith : How to be classy, how to be cute, how to actually walk the walk... Love you for this @Cloudflare ❤️ https://www.nytimes.com/... LinkedIn: Kate Knibbs : NEW: Cloudflare will now block AI bots by default. It's also launching a “Pay Per Crawl” options so users can haggle with AI companies if they want. … Sam Jacobs : Happy to see TIME join this initiative with Cloudflare to protect journalism. — “Mark Howard, the chief operating officer of TIME, said he welcomed Cloudflare's move. … Björn Rudling : For everyone in the publishing space and content creation, look no further. — Cloudflare is changing the way the internet works. … Forums: r/selfhosted : Cloudflare will now block AI crawlers by default r/webscraping : Cloudflare to introduce pay-per-crawl for AI bots r/CloudFlare : Introducing pay per crawl: enabling content owners to charge AI crawlers for access r/technology : Cloudflare launches a marketplace that lets websites charge AI bots for scraping See also Mediagazer
Context & Ripple Effects
Cloudflare had already moved from passive web delivery into crawler governance with its tool for blocking AI-training bots. Publishers had also tested bot-specific restrictions in robots.txt, while reports of Anthropic using newly named bots exposed the limits of name-based blocking alone.
This release turns that defensive posture into a commercial control point: access can be denied by default or priced through Cloudflare’s network. The private beta matters because it tests whether publishers and creators can convert crawler access into a usable market rather than merely an opt-out mechanism.
First-order effects
- New Cloudflare sites will block AI crawlers unless the site owner opts in, immediately shifting the default toward publisher permission.
- Participating publishers and creators can set a per-crawl charge through Pay per Crawl, while AI crawlers seeking that content face a new access-and-payment decision.
Second-order effects
- AI companies will need to manage a more fragmented set of site permissions and potential access fees, rather than relying solely on crawler exclusions such as the GPTBot opt-out approach.
- Cloudflare gains a role as both enforcement layer and transaction intermediary; its early participants may influence whether per-crawl pricing becomes a practical publisher control or remains a limited beta feature.
Third-order effects
- If adoption broadens, web content could increasingly be treated as metered input for AI systems, shifting leverage from unilateral crawling toward negotiated access and licensing.
- The move also points to an infrastructure-centered market for AI access controls, though its durability depends on crawler compliance and whether enough content owners and AI firms participate.
The trend: AI content access is moving from voluntary crawler exclusions toward platform-enforced permissions and monetization.