Amazon blames a rare software bug and “faulty automation” for this week's massive AWS glitch, and says it has turned off the flawed automation worldwide
In-depth Amazon coverage from the tech giant's hometown, including e-commerce, AWS, Amazon Prime, Alexa, logistics, devices, and more.
GeekWireTodd Bishop
Context & Ripple Effects
AWS has repeatedly had to explain broad service disruptions: a prior incident was attributed to overloaded network devices and prompted planned changes to its status communications, while another outage halted Amazon’s Flex and Dolphin logistics and time-tracking apps.
This incident matters because Amazon is responding at the control layer, disabling the implicated automation worldwide rather than treating the glitch as a contained regional failure.
First-order effects
AWS customers and Amazon-operated services affected by the glitch face recovery from the immediate disruption while AWS removes the flawed automation from production globally.
AWS engineering and operations teams must replace or constrain the disabled automation and validate that the mitigation does not introduce new manual operational gaps.
Second-order effects
The worldwide shutdown raises the operational burden of running affected AWS workflows, likely increasing internal review of automated changes and rollback procedures.
Customers whose applications depend on AWS availability have a fresh reason to test failover and incident-response plans, following earlier outages that also affected external sites and services.
Third-order effects
If automation-related incidents recur, cloud reliability will increasingly depend on how providers govern automated control paths—not only on the resilience of underlying compute and network hardware.
The pattern points toward more explicit guardrails, observability, and human approval around high-impact infrastructure automation, though this incident alone does not establish a lasting policy change.
The trend: Cloud platforms are making operational automation more central to service delivery, increasing the importance of safeguards around automation that can affect shared infrastructure at scale.
“The internet was designed to be resilient; many other channels existed for routing around problems or attacks, but we've lost some of that resilience by becoming so dependent on a handful of giant tech companies to provide not just data storage but also house data services.”
It's been pretty funny watching the media class and the bourgeoisie wailing and gnashing their teeth over this. It had zero effect on me and I'm still not quite sure what exactly the problem was, but then again, I'm not the sort of dickhead that has a bed that's connected to the…
Report on what caused the AWS incident is out. — It's a good read, accessible to most people I think if you're interested. — As always with these things, one werid thing happened and then a lot of other stuff backed up.
The last time this happened a person loaded a file with one typo and the outage was way bigger. — But look thankfully there are like 5 services versus 3 then. [embedded post]
“flawed automation” is the new term for AI Agent. Now some poor AWS IT person is gonna get paged at 3 am every Monday morning to do the manual work the flawed automation couldn't handle. [embedded post]
Eight Sleep owners were shocked to find they could not adjust their beds using the app when AWS was down. The company has now put out an update that uses Bluetooth so beds won't be stuck in an upright position because you were reading in bed whenever there's an AWS outage. — W…
Excellent post-mortem the AWS us-east-1 region failure this week: — https://aws.amazon.com/... Tl;dr: — 1. Consistency and concurrency control are hard in distributed database systems. Amazon's DynamoDB discovered a new corner case that kicked the whole mess off. …