Microsoft says “Bing stopped crawling Reddit” after Reddit updated its robots.txt file on July 1 to prohibit “all crawling of their site”
Reddit has updated its robots.txt file, preventing Bing and many other search engines from crawling the site.
Context & Ripple Effects
Reddit had already signaled a robots.txt revision to require automated agents to follow its terms. The July change turns that policy signal into an operational restriction for Bing and other crawlers.
The effect is visible in contemporaneous reporting that several non-Google search engines lacked recent Reddit results, making a publisher-level access rule consequential for how discussion content is discovered across search products.
First-order effects
- Bing can no longer crawl Reddit for newly discoverable or refreshed material while the restriction remains in place, limiting the freshness of Reddit results in its index.
- Reddit gains a direct technical control over which automated services can collect its pages, rather than relying only on stated usage terms.
Second-order effects
- Search services affected by the block must either accept less current Reddit coverage or seek an arrangement that restores access; users may see different Reddit result availability across engines.
- The change strengthens Reddit's leverage in discussions with companies that want its data, because access to current content is no longer assumed.
Third-order effects
- If more large publishers use crawler controls this way, web-search indexes may become less uniform and access to timely public-web content may increasingly depend on publisher-platform agreements.
- The boundary between conventional search crawling and AI-data collection is likely to remain contested, with robots.txt becoming a practical bargaining mechanism rather than merely a technical preference.
The trend: Publisher controls over AI and search crawling are shifting access to high-value web content from open indexing toward negotiated terms.