Facebook says configuration changes on its backbone routers disrupted network traffic between its data centers, causing an outage across its platforms

To all the people and businesses around the world who depend on us, we are sorry for the inconvenience caused by today's outage across our platforms.

Facebook Engineering 2021-10-05 Santosh Janardhan

Context & Ripple Effects

Facebook’s incident follows a 2019 service disruption also attributed to a server configuration change, making configuration safety a recurring operational issue in the company’s coverage. The immediate report identifies the shared backbone between data centers as the failure domain rather than an isolated app problem.

Facebook’s subsequent account narrowed the trigger to a command used to assess global backbone capacity, explaining why recovery required restoring underlying network connectivity before its services could return.

First-order effects

Facebook’s platforms lose the data-center connectivity required to serve users and businesses, while its engineering teams must restore and validate backbone routing before application-level recovery can proceed.

Second-order effects

A single backbone change taking multiple Facebook services down raises the operational cost of network-maintenance commands, pushing Facebook toward tighter validation and rollback controls for capacity-assessment work.

Third-order effects

Repeated configuration-linked outages point to a structural reliability challenge for tightly integrated platforms: shared infrastructure can deliver scale but also concentrates failures across many user-facing services.

The trend: Large internet platforms are treating backbone configuration management as a core resilience problem because shared network layers create broad failure domains.

Discussion

@alexhern Alex Hern on x
Facebook (accidentally, we assume) sent an update to a deep-level routing protocol on the internet that said, basically, “hey we don't have any servers any more xoxo”
@alexhern Alex Hern on x
but Facebook runs EVERYTHING through Facebook
@alexhern Alex Hern on x
Normally, this would be quite easy to fix. you just send another update saying “oh, don't worry, we have servers, they're here, xoxo”. Things still break, it takes a while for the message to spread to all corners of the internet, egg on face, but liveable
@jdan Jordan Scales on x
Absolutely wild insider info from a friend who works at facebook (name and photo hidden obviously) https://twitter.com/...
@copyconstruct Cindy Sridharan on x
Yep, a configuration change indeed! A mini postmortem of today's Facebook outage is now up. https://engineering.fb.com/... https://twitter.com/... https://twitter.com/...
@donie Donie O'Sullivan on x
“We also have no evidence that user data was compromised as a result of this downtime.”
@briankrebs @briankrebs on x
“We want to make clear at this time we believe the root cause of this outage was a faulty configuration change. We also have no evidence that user data was compromised as a result of this downtime.”
@nick_craver Nick Craver on x
@fb_engineering Will there be a in-depth public postmortem so others can learn and improve from today's outage?
@conormyhrvold @conormyhrvold on x
Also, props for authorship and having an infra exec take ownership of the outage. Never an easy thing to do, and would've been an easy out to have it come from some generic The Facebook Team or something like that.
@briankrebs @briankrebs on x
FB: Configuration changes on backbone routers that coordinate network traffic between our data centers caused issues that interrupted this communication. This had a cascading effect on the way our data centers communicate, bringing our services to a halt. https://engineering.fb.c…
@carnage4life Dare Obasanjo on x
Facebook engineers on today's outage: Configuration changes on the backbone routers that coordinate network traffic between data centers caused the issue. This disruption to network traffic had a cascading effect on the way the data centers communicate. https://engineering.fb.com…
@dnsmichi @dnsmichi on x
Interesting insight into Facebook's BGP implementation. https://engineering.fb.com/... https://twitter.com/...
@bgurley Bill Gurley on x
We now know which platform you go to if there is something super important to convey. $TWTR https://twitter.com/...

Chronicles