Cloudflare says the November 18 outage was caused by a change to one of its “database systems' permissions”, not by “malicious activity of any kind”

On 18 November 2025 at 11:20 UTC (all times in this blog are UTC), Cloudflare's network began experiencing significant failures to deliver core network traffic.

Cloudflare 2025-11-19 Matthew Prince

Context & Ripple Effects

Cloudflare had already said the service disruption was resolved on the day it occurred; this account identifies the internal change behind that event rather than treating it as an external attack. It follows a prior Cloudflare incident in which a bug caused customer log-delivery losses, underscoring that operational defects can affect more than one layer of the platform.

The explicit denial of malicious activity also distinguishes this episode from Cloudflare’s disclosed 2023 intrusion tied to stolen Okta tokens. For customers, the material issue is therefore the resilience of production access controls and change processes.

First-order effects

Cloudflare’s core-traffic delivery failed during the incident, directly disrupting customers and the services that depend on its network.
Attributing the event to a database-permission change narrows the immediate response toward reviewing and remediating internal configuration and authorization controls, rather than activating an intrusion-led response.

Second-order effects

Customers that rely on Cloudflare for critical delivery paths are likely to reassess failover plans and the operational assurances they require from the provider after the same-day resolution notice.
The incident raises the value of safeguards around high-impact permission changes—such as staged rollout, validation, and rapid rollback—across infrastructure providers with shared control planes.

Third-order effects

If similar incidents recur, cloud-edge providers will face growing pressure to treat administrative permissions as production-critical reliability surfaces, not merely security settings.
The broader risk is concentration of operational blast radius: a small internal control-plane change can interrupt many downstream services at once, increasing demand for independently testable resilience and multi-provider contingency options.

The trend: This is part of a broader shift in which infrastructure reliability increasingly depends on governing internal configuration and identity changes with the same rigor as external cyber threats.

Discussion

@eastdakota Matthew Prince on x
We let the Internet down today. Here's our technical post mortem on what happened. On behalf of the entire @Cloudflare team, I'm sorry. https://blog.cloudflare.com/ ...
@olmec_dongdold @olmec_dongdold on x
It shouldn't be possible for you to “let down the internet”. It's completely absurd to have global infrastructure at the mercy of a single point of failure
@charles_irl @charles_irl on x
https://blog.cloudflare.com/ ... oh my god, it was an ML feature pipeline [image]
@dani_avila7 Daniel San on x
Really appreciate Cloudflare's transparency here. This is how you write a postmortem: They had a 3hr outage today caused by a database permissions change that doubled their Bot Management config file size, hitting a hardcoded limit in their Rust proxy. The config regenerated
@niccoatesuk Nic Coates on x
Incidents happen. But providing a detailed RCA helps regain trust. Nice one @Cloudflare team. (Suggest one other follow up action: review your status page provider and make sure it's remains up during an incident 😉)
@gergelyorosz Gergely Orosz on x
More like: September 26: got new running shoes because the previous one was worn down and falling apart 53 days later: fell flat on my face while running my usual route, tripping on something unexpected “The new shoe is clearly to blame” (clearly it's not)
@cyb3rops Florian Roth on x
Cloudflare went down because a ClickHouse change made a metadata query return duplicate columns. That blew up the size of a config file, it spread across the network and one module couldn't handle the larger file and crashed. Everything after that was basically fallout from this
@tannerlinsley Tanner Linsley on x
This is one of the best postmortems I've ever read.
@wishee0 Vaish on x
sooo, @Cloudflare happened to take down half the internet cause... a file was too big??? - db permissions change at 11:05 made a query return duplicate rows (default + r0 db) - bot management feature file doubled from ~60 to 200+ entries - rust code had hardcoded 200 feature [ima…
@vicvijayakumar Vic on x
lost a lot of money today betting on dns [image]
@uk_daniel_card @uk_daniel_card on x
The cloudflare report is good to see https://blog.cloudflare.com/ ... [image]
@jondot @jondot on x
TLDR; A single unwrap() dropped the Internet. This is not Rust at fault, it's similar to any give NPE in any given other language only that with Rust you are forced to decide what to do. The engineer in this case decided that it's OK to crash. Mistakes in judgement can happen
@timurguvenkaya Timur Guvenkaya on x
The internet was down because of the basic panic on .unwrap() in Rust Never use unwrap/expect in production unless: a) It is in tests b) You have safeguards before c) It is asserting initial system state [image]
@malwarejake Jake Williams on x
The Cloudflare outage was caused by a configuration file growing too large. Likely, they were blocking too many scrapers and caused out of memory exceptions parsing the file. I'll be amazed if blocking AI scrapers isn't the outage catalyst. https://www.theverge.com/...
@jacksh4h Jack on x
@eastdakota @Cloudflare McD down how to eat Big Mac? [image]
@lundukejournal @lundukejournal on x
September, 26: Cloudflare rewritten in “memory safe” Rust. The change is touted as “faster and more secure” because of Rust. https://blog.cloudflare.com/ ... November, 18 (53 days later): Cloudflare has a massive outage, which took down large portions of the Internet, because of …
@gergelyorosz Gergely Orosz on x
Imagine if all major tech companies took outages as seriously, and held themselves accountable publicly as Cloudflare does. But 99% of them do not: and so I trust Cloudflare more than almost any other company (including the hyperscalers). Cloudflare next level in transparency
@eastdakota Matthew Prince on x
@DCathal @Cloudflare Hardly had time to wordsmith. Think record for time from major incident to full technical post mortem. I may be listed as the author but took a whole team across the org to pull together the details to write.
@eastdakota Matthew Prince on x
@Anubhav_Mishra_ @Cloudflare I disagree. We can be better. By far.
@_felipe Felipe O. Carvalho on x
We need an .unwrap_in_debug_builds_but_just_ carry_on_in_release_builds() function in Rust to stop these incidents. I have recently caused problems with a panic that would be informative in debug builds during testing but didn't really have to fire in production. [image]
@shrirambalaji Shriram Balaji on x
wild, a single unwrap caused panic, and led to half the Internet going down https://github.com/..., we probably need a lint rule or something to avoid unwrap leaking into systems that shouldn't fail [image]
@ittaia Ittai Abraham on x
Wonderful postmortem! Configuration change of a database you didn't know about renders internet wide liveness failure. 1987: A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable https://lamport.azurewebs…
@cloudflare @cloudflare on x
On November 18 Cloudflare experienced a service outage, triggered by an issue with a Bot Management feature, impacting multiple Cloudflare services. Here's a detailed breakdown of what happened. https://blog.cloudflare.com/ ...
r/sre r on reddit
Cloudflare Outage Postmortem
r/programming r on reddit
Cloudflare outage on November 18, 2025 - official response
r/Futurology r on reddit
Internet backbone failure raises questions about centralized cloud infrastructure

Chronicles