OpenAI's crawlers took down e-commerce site Triplegangers by relentlessly trying to scrape the entire site, whose robots.txt file was not properly configured
techcrunch.com/2025/01/10/h... #google #seo #openai [image] @tante.cc : #OpenAI is basically the locusts of the digital by now. Their massive scrapers crushing websites in order to steal and feed their bullshit generators. — techcrunch.com/2025/01/10/h... @valkayec : I don't know, scraping an entire site & causing it to go down feels kinda wrong. Kinda like stealing something and breaking it. [embedded post] Glyn Moody / @glynmoody : How OpenAI's bot crushed this seven-person company's website ‘like a DDoS attack’ - techcrunch.com/2025/01/10/h... “you can opt out if you update your robot.txt with our tags,” says Tomchuk, but that puts the onus on the business owner to understand how to block them. Mastodon: Dr Pen / @DrPen@mastodon.social : Required reading for everyone to know just what you're dealing with when the AI bots come crawling. Update your robots txt now and use other machine readable methods in addition (metadata, htaccess, exif etc) — #ai #openai — How OpenAI's bot crushed this seven-person company's website ‘like a DDoS attack’ | … Miguel Afonso Caetano / @remixtures@tldr.nettime.org : “On Saturday, Triplegangers CEO Oleksandr Tomchuk was alerted that his company's e-commerce site was down. It looked to be some kind of distributed denial-of-service attack. — He soon discovered the culprit was a bot from OpenAI that was relentlessly attempting to scrape his entire, enormous site. … @josejfernandez@todon.eu : How OpenAI's bot crushed this seven-person company's website ‘like a DDoS attack’ — https://techcrunch.com/... It could have been prevented by using robots.txt ( https://en.wikipedia.org/... BUT: — It is an OPT-IN system. All undesired AI scrappers would need to be manually added. … Forums: Hacker News : OpenAI's bot crushed this seven-person company's web site ‘like a DDoS attack’ BeauHD / Slashdot : OpenAI's Bot Crushes Seven-Person Company's Website ‘Like a DDoS Attack’
Context & Ripple Effects
The incident exposes the practical limits of an opt-out model: OpenAI had described GPTBot controls through robots.txt, but Triplegangers' misconfiguration left a small commerce site bearing the operational burden of enforcement.
It follows a wider turn toward defensive web controls, including publishers changing robots.txt to block Anthropic bots. The concern is no longer solely whether training crawls are permitted, but whether their traffic can be safely absorbed.
First-order effects
- Triplegangers faces immediate availability and potential sales disruption after OpenAI crawler traffic overwhelmed the site in a DDoS-like outage.
- Site operators must correctly configure robots.txt and monitor crawler traffic if they want to prevent or constrain automated collection under the current opt-out approach.
Second-order effects
- Smaller publishers and e-commerce operators may add bot mitigation or hosting capacity, shifting the cost of AI-data collection from crawler operators to site owners.
- Infrastructure providers gain a clearer role as an enforcement layer as sites seek protections beyond robots.txt, a direction reflected in tools designed to challenge or misdirect AI scrapers.
Third-order effects
- If high-volume AI crawling repeatedly strains origin sites, robots.txt alone may prove insufficient as a consent mechanism, increasing pressure for crawler identity, rate limits, and more enforceable access controls.
- The web could become more segmented between operators able to deploy bot defenses and those that cannot, making access infrastructure a more consequential gatekeeper for AI training data.
The trend: AI training-data collection is shifting from a permissions debate toward an infrastructure-control problem in which website operators need practical ways to manage automated demand.