Reddit says it will update its robots.txt to make “as clear as possible” that companies “using an automated agent to access Reddit” need to abide by its terms
The warning comes after reports that AI companies regularly ignore instructions not to scrape.
Reddit plans to update its robots.txt file to more clearly ban AI bots. The problem is there isn't a clear legal framework for blocking scraping. Some scrapers have won lawsuits on the basis the info is public while others have lost for violating the site's terms of service. [ima…
The sticking point on legality of scraping tends to rest on whether the data in question requires a login to access or is public. The game of cat and mouse is on. https://www.engadget.com/...