OpenAI's crawlers took down e-commerce site Triplegangers by relentlessly trying to scrape the entire site, whose robots.txt file was not properly configured
On Saturday, Triplegangers CEO Oleksandr Tomchuk was alerted that his company's e-commerce site was down. Bluesky: @valkayec , @glynmoody , and @tante.cc Mastodon: @remixtures@tldr.nettime.org , @DrPen@mastodon.social , and @josejfernandez@todon.eu Forums: Hacker News and Slashdot Bluesky: @valkayec : I don't know, scraping an entire site & causing it to go down feels kinda wrong. Kinda like stealing something and breaking it. [embedded post] Glyn Moody / @glynmoody : How OpenAI's bot crushed this seven-person company's website ‘like a DDoS attack’ - techcrunch.com/2025/01/10/h... “you can opt out if you update your robot.txt with our tags,” says Tomchuk, but that puts the onus on the business owner to understand how to block them. @tante.cc : #OpenAI is basically the locusts of the digital by now. Their massive scrapers crushing websites in order to steal and feed their bullshit generators. — techcrunch.com/2025/01/10/h... Mastodon: Miguel Afonso Caetano / @remixtures@tldr.nettime.org : “On Saturday, Triplegangers CEO Oleksandr Tomchuk was alerted that his company's e-commerce site was down. It looked to be some kind of distributed denial-of-service attack. — He soon discovered the culprit was a bot from OpenAI that was relentlessly attempting to scrape his entire, enormous site. … Dr Pen / @DrPen@mastodon.social : Required reading for everyone to know just what you're dealing with when the AI bots come crawling. Update your robots txt now and use other machine readable methods in addition (metadata, htaccess, exif etc) — #ai #openai — How OpenAI's bot crushed this seven-person company's website ‘like a DDoS attack’ | … @josejfernandez@todon.eu : How OpenAI's bot crushed this seven-person company's website ‘like a DDoS attack’ — https://techcrunch.com/... It could have been prevented by using robots.txt ( https://en.wikipedia.org/... BUT: — It is an OPT-IN system. All undesired AI scrappers would need to be manually added. … Forums: Hacker News : OpenAI's bot crushed this seven-person company's web site ‘like a DDoS attack’ BeauHD / Slashdot : OpenAI's Bot Crushes Seven-Person Company's Website ‘Like a DDoS Attack’ 2
Context & Ripple Effects
OpenAI had previously described GPTBot and a robots.txt-based opt-out path in its GPTBot crawler documentation, placing responsibility for blocking collection largely with site operators.
That approach was already under strain as major publishers revised robots.txt rules to block AI bots. Triplegangers shows that, for a small e-commerce operator, a configuration gap can become an availability problem rather than only a data-use dispute.
First-order effects
- Triplegangers loses site availability while its team must correct crawler controls and restore normal access for shoppers.
- The incident highlights that OpenAI's crawler can impose substantial load when a site has not configured its robots.txt protections correctly.
Second-order effects
- Smaller web operators have a stronger incentive to pair robots.txt with traffic controls and bot mitigation, rather than treating the opt-out file as sufficient protection.
- AI crawler operators face added pressure to make crawl behavior more conservative around smaller sites, because disruption turns a content-access issue into an operational-risk issue.
Third-order effects
- If similar incidents persist, robots.txt is likely to become only one layer in AI-data governance, supplemented by active rate limiting, challenge systems, and infrastructure-level blocking.
- The web's open crawling model may increasingly split between sites able to absorb or manage automated traffic and those that need intermediaries to protect availability.
The trend: AI training crawlers are pushing website owners from voluntary indexing preferences toward enforceable controls over automated access and server capacity.