Reddit CEO Steve Huffman says Microsoft, Anthropic, and Perplexity need to pay to scrape Reddit's data, and blocking them has been “a real pain in the ass”
After striking deals with Google and OpenAI, Reddit CEO Steve Huffman is calling on Microsoft and others to pay if they want to continue scraping the site's data.
Context & Ripple Effects
Reddit’s position extends its earlier argument that it could not subsidize commercial use of its API and that its API was not built to support third-party apps. The company is now applying that boundary to AI and search-data collection rather than only conventional API clients.
The demand also follows reports that Reddit was restricting recent results from search engines outside Google’s indexing path, while Google and OpenAI had already secured licensing deals. That makes data access a negotiated commercial relationship, not simply an open-web default.
First-order effects
- Microsoft, Anthropic, and Perplexity face a choice between negotiating paid access to Reddit data or losing a source they had been scraping.
- Reddit must devote engineering and operational effort to enforcing blocks, even as it seeks to convert access into licensing revenue.
Second-order effects
- Licensed partners gain a clearer access advantage over firms that rely on unauthorised crawling, increasing pressure on rival AI and search providers to strike comparable deals.
- Blocking friction exposes the limits of technical access controls: platforms may need a mix of crawler restrictions, contracts, and ongoing enforcement rather than a one-time policy change.
Third-order effects
- If more platforms follow Reddit’s approach, high-value conversational and community content could shift from broadly crawlable web material to licensed input for AI products.
- The resulting market may favor large platforms able to negotiate and enforce access terms, while smaller publishers face a harder trade-off between search visibility and control over AI use.
The trend: AI companies’ demand for fresh web content is pushing publishers and platforms to replace passive crawling with paid, controlled data access.