Facebook says a command issued to assess the availability of global backbone capacity caused its outage, in a post offering more details on its recovery efforts
Facebook had initially attributed the disruption to configuration changes on backbone routers that interrupted traffic among its data centers. Cloudflare separately observed BGP updates and ensuing DNS-resolution failures, placing the outage at the network-control layer rather than within an individual app.
The new account narrows that chain further: a command intended to assess global backbone capacity was the initiating action. That matters because the recovery account turns a broad configuration failure into a specific example of operational tooling affecting shared infrastructure.
First-order effects
Facebook’s engineering account now ties the outage to a capacity-assessment command, giving its recovery effort a more defined failure mode than the initial router-configuration explanation.
Facebook, Instagram, WhatsApp, and Messenger share exposure to backbone-control failures; a prior outage had already affected that same group of services in tandem.
Second-order effects
Cloudflare’s BGP and DNS observations are more readily explained as downstream effects of a backbone-control failure, strengthening the case that external reachability can fail alongside Facebook’s internal connectivity.
The incident makes diagnostic activity itself part of Facebook’s production-risk surface, alongside the configuration changes implicated in earlier outages.
Third-order effects
Repeated configuration-linked disruptions point toward a reliability model in which management-plane commands receive the same safeguards as customer-facing production changes, because shared backbone infrastructure can create correlated failures across Facebook’s service portfolio.
The trend: Large internet platforms are treating backbone operations and diagnostic tooling as high-blast-radius production systems rather than isolated infrastructure administration.
“Individual data centers were reporting dips in power usage in the range of tens of megawatts” be interesting to see the effect of this outage on the electricity grid for the time it was down https://twitter.com/...
Yesterday's outage across our products was a bad one, so we're sharing some more detail here on exactly what happened, how it happened, and what we're learning from it: https://engineering.fb.com/...
Facebook's data centres specifically had a serious power demand / supply issue. Their load is so high that suddenly re-activating their service could have caused grid instability: https://engineering.fb.com/... https://twitter.com/...
On today's episode of Cascading Failures - “Our systems are designed to audit commands like these to prevent mistakes like this, but a bug in that audit tool didn't properly stop the command” https://twitter.com/...
Didn't think we'd hear much of anything so pleasantly surprised - a very nice debrief in just the right amount of detail. Some interesting things to ponder on the nature of very large global services, security vs. maintainability, etc. https://twitter.com/...
I could completely understand shutting an interface or two on a few devices, but I'm pretty sure if an engineer was to type out all of Facebooks device and interface parings they would be there a long time.
If we accept this explanation at face value it means that there is a serious flaw in a critical system (tool) that has been overlooked which shouldn't be the case in such a global scale platform nor in engineering culture supposedly mature. https://twitter.com/...
Very interesting read, there's much to learn. Even when you're prepared to everything that could come in (very smart people's) mind, this kind of outage can happen. It reminds me plane and nuclear industries, very mature ones today. Datacenter industry is still learning https://t…