2024-07-25
Search Engine Land
8 related
Microsoft says “Bing stopped crawling Reddit” after Reddit updated its robots.txt file on July 1 to prohibit “all crawling of their site”
Reddit has updated its robots.txt file, preventing Bing and many other search engines from crawling the site.
2024-06-26
Engadget
7 related
Reddit says it will update its robots.txt to make “as clear as possible” that companies “using an automated agent to access Reddit” need to abide by its terms
The warning comes after reports that AI companies regularly ignore instructions not to scrape.
2021-03-23
Wall Street Journal
Analysis of 3,100 sites shows that hospitals are using code to hide pages containing pricing data from being crawled by search engines, violating US federal law
Webpages for hundreds of hospitals require users to click through to find prices, undermining federal transparency rule, Journal analysis shows Tweets: @ericgeller , @annemariebridy , @sarahkliff , @d...
Loading articles...