Cloudflare CEO Matthew Prince says Google's ratio of pages crawled per visitor sent to a publisher fell from 2:1 10 years ago to 18:1; for OpenAI, it's 1,500:1
Publishers face an existential threat in the AI era and need to take action to make sure they are fairly compensated for their content …
Context & Ripple Effects
Matthew Prince’s crawl-to-referral figures put a concrete ratio on publishers’ growing complaint that AI-mediated discovery extracts content without returning comparable audience value. That concern had already surfaced in reports of traffic declines after Google AI Overviews.
The imbalance also helps explain Cloudflare’s subsequent shift toward site-level controls and commercialization, including its Pay per Crawl marketplace and default AI-crawler blocking for new sites.
First-order effects
- Publishers using Cloudflare gain a clearer benchmark for deciding whether to permit, restrict, or charge AI crawlers, especially where crawler activity far exceeds referred visitors.
- Google and OpenAI face sharper publisher scrutiny over whether crawling terms and referral economics adequately compensate the sites supplying source material.
Second-order effects
- Crawler access becomes a negotiation point rather than a passive technical default, increasing the value of products that meter, block, or specify permitted uses of publisher content.
- Search and chatbot providers may face a more fragmented content supply as publishers adopt controls; Cloudflare later expanded this approach through a Content Signals Policy for crawler-use preferences.
Third-order effects
- The web’s historical exchange of crawl access for referral traffic could be replaced by differentiated access rules, licensing, or per-use pricing where AI answers reduce the value of the referral.
- If publishers cannot recover value through either traffic or payments, AI distribution could concentrate bargaining power with the platforms that control user interfaces and crawler access.
The trend: AI answer products are turning publisher content from an advertising-supported discovery asset into an input whose access, attribution, and compensation are increasingly negotiated.