OpenAI's crawlers took down e-commerce site Triplegangers by relentlessly trying to scrape the entire site, whose robots.txt file was not properly configured
On Saturday, Triplegangers CEO Oleksandr Tomchuk was alerted that his company's e-commerce site was down. Bluesky: @valkayec , @glynmoody , and @tante.cc Mastodon: @remixtures@tldr.nettime.org , @DrPen@mastodon.social , and @josejfernandez@todon.eu Forums: Hacker News and Slashdot Bluesky: @valkayec : I don't know, scraping an entire site & causing it to go down feels kinda wrong. Kinda like stealing something and breaking it. [embedded post] Glyn Moody / @glynmoody : How OpenAI's bot crushed this seven-person company's website ‘like a DDoS attack’ - techcrunch.com/2025/01/10/h... “you can opt out if you update your robot.txt with our tags,” says Tomchuk, but that puts the onus on the business owner to understand how to block them. @tante.cc : #OpenAI is basically the locusts of the digital by now. Their massive scrapers crushing websites in order to steal and feed their bullshit generators. — techcrunch.com/2025/01/10/h... Mastodon: Miguel Afonso Caetano / @remixtures@tldr.nettime.org : “On Saturday, Triplegangers CEO Oleksandr Tomchuk was alerted that his company's e-commerce site was down. It looked to be some kind of distributed denial-of-service attack. — He soon discovered the culprit was a bot from OpenAI that was relentlessly attempting to scrape his entire, enormous site. … Dr Pen / @DrPen@mastodon.social : Required reading for everyone to know just what you're dealing with when the AI bots come crawling. Update your robots txt now and use other machine readable methods in addition (metadata, htaccess, exif etc) — #ai #openai — How OpenAI's bot crushed this seven-person company's website ‘like a DDoS attack’ | … @josejfernandez@todon.eu : How OpenAI's bot crushed this seven-person company's website ‘like a DDoS attack’ — https://techcrunch.com/... It could have been prevented by using robots.txt ( https://en.wikipedia.org/... BUT: — It is an OPT-IN system. All undesired AI scrappers would need to be manually added. … Forums: Hacker News : OpenAI's bot crushed this seven-person company's web site ‘like a DDoS attack’ BeauHD / Slashdot : OpenAI's Bot Crushes Seven-Person Company's Website ‘Like a DDoS Attack’ 2