Cloudflare says prior to Facebook's outage, Facebook issued several BGP updates, causing DNS resolution errors that knocked Facebook off the web
“Facebook can't be down, can it?”, we thought, for a second. — Today at 15:51 UTC, we opened an internal incident entitled …
The Cloudflare BlogCelso Martinho
Context & Ripple Effects
Cloudflare’s routing view identifies the outward-facing failure mode, while Facebook’s same-day account tied the disruption to configuration changes on backbone routers. The two accounts place DNS reachability and inter-data-center traffic in the same incident chain.
Facebook later said a capacity-assessment command triggered the outage, underscoring how a backbone change propagated beyond internal network operations to public service discovery.
First-order effects
Facebook’s BGP changes produced DNS resolution errors, preventing users and external systems from reaching Facebook services even though the failure originated in its network backbone.
Cloudflare could expose the routing and DNS symptoms externally, giving the incident a network-level diagnosis alongside Facebook’s internal explanation.
Second-order effects
Facebook’s recovery process had to address both backbone traffic disruption and the public DNS reachability failures that made its services unavailable from outside its network.
The episode gives network operators a concrete reason to treat routing changes as changes to customer-facing service availability, not solely as internal capacity operations.
Third-order effects
If backbone operations increasingly depend on globally propagated routing changes, large platforms will need incident controls that test both internal traffic paths and external DNS visibility before changes reach production.
The incident points toward greater reliance on independent network telemetry during major outages, because external observers can identify reachability failures while an affected operator is still isolating the initiating change.
The trend: Internet-scale service reliability is increasingly governed by the interaction between private backbone changes and public routing and DNS systems.
Confirmed: The DNS records that tell systems how to find https://facebook.com/ or https://instagram.com/ got withdrawn this morning from the global routing tables. Can you imagine working at FB right now, when your email no longer works & all your internal FB-based tools fail?
@briankrebs
@briankrebs on x
From trusted source: Person on FB recovery effort said the outage was from a routine BGP update gone wrong. But the update blocked remote users from reverting changes, and people with physical access didn't have network/logical access. So blocked at both ends from reversing it.
@evacide
Eva on x
October is now BGP Awareness Month.
@kantrowitz
Alex Kantrowitz on x
Zuck weighs in on the outage: “Sorry for the disruption today — I know how much you rely on our services to stay connected with the people you care about.” https://twitter.com/...
@eastdakota
Matthew Prince on x
Here's our best explanation from what we can see on how @Facebook disappeared from the Internet: https://blog.cloudflare.com/ ...
@cloudflare
@cloudflare on x
“Facebook can't be down, can it?”, we thought, for a second. Well it can, and here's how. https://blog.cloudflare.com/ ...
@craigthomler
Craig Thomler on x
@kcarruthers Technically it was BGP, but impact was mainly DNS. Essentially this emphasizes the importance of accurate maps in digital space as well as in physical space. https://blog.cloudflare.com/ ...
@eastdakota
@eastdakota on x
Nothing we're seeing related to the Facebook services outage suggests it was an attack. Most likely explanation is that the company's Internet routes (BGP) were withdrawn by mistake during maintenance. #hugops
@zsk
Zoe Kleinman on x
Good explanation here on the Facebook fiasco from Cloudflare https://twitter.com/...
@gte
Guy English on x
Our long national nightmare has been restored.
@fooinha
Paulo Pacheco on x
Great blog post from @Cloudflare and how they managed to keep their services running smoothly despite “a tsunami of additional DNS traffic” following Facebook's authoritative servers DNS outage “Understanding How Facebook Disappeared from the Internet” https://blog.cloudflare.com…
@thingskatedid
Kate on x
BGP is one of those protocols nobody appreciates until it stops working
@davidgerard
@davidgerard on x
cloudflare blogged about the Facebook outage, pretty good explanation, including of BGP (the thing FB screwed up): https://blog.cloudflare.com/ ...
@troyhunt
Troy Hunt on x
It's always DNS... “Understanding How Facebook Disappeared from the Internet” https://blog.cloudflare.com/ ...
@briankrebs Cloudflare has a good explainer on what appears to have happened. What is crazy is that they didn't appear to have an out-of-band management channel in place to reconfigure routers - and needed to go onsite to reconfigure. https://blog.cloudflare.com/ ...
@katebevan
Kate Bevan on x
This is a bit nerdy but pretty accessible explanation of what's happened. Kudos to Cloudflare for pulling this explainer together and sharing it so quickly https://twitter.com/...
@davidgerard
@davidgerard on x
from NANOG, on the Facebook outage: they meant to drop BGP peering to one network, accidentally dropped *all* of them, and now can't get back in to un-drop them. Other difficulties. This is a normal IT fuckup, just a very public and spectacular one. https://mailman.nanog.org/...