CrowdStrike shares a root cause analysis of the July 19 outage, and says it hired two third-party security firms to review its Falcon threat-detection suite
And reveals the small mistake that bricked 8.5M Windows boxes — CrowdStrike has hired two outside security firms to review …
The Register Jessica Lyons
Related Coverage
- External Technical Root Cause Analysis — Channel File 291 CrowdStrike
- View article Cybersecurity Dive
- View article SC Media
- CrowdStrike Releases Root Cause Analysis Of Falcon Sensor BSOD Crash SecurityWeek · Ryan Naraine
- CrowdStrike Investigates IT Outage, Implements New Safeguards WinBuzzer · Luke Jones
- CrowdStrike Reveals Root Cause of Global System Outages The Hacker News
- CrowdStrike Releases Outage Root Cause Analysis as Legal Battle Heats Up The Cyber Express · Paul Shread
- CrowdStrike reveals ‘root cause’ of global Microsoft meltdown The Independent · Vishwam Sankaran
- CrowdStrike explains root cause of its giant IT outage. The Verge · Tom Warren
- CrowdStrike explains how it BSOD'd the world Runtime · Tom Krazit
- CrowdStrike: More Testing, Staged Rollouts Now In Place For Updates CRN · Kyle Alspach
- Things we'll now check at compile time: — “The new IPC Template Type defined 21 input parameter fields, but the integration code that invoked the Content Interpreter with Channel File 291's Template Instances supplied only 20 input values to match against.” — https://www.crowdstrike.com/ ... @gregeganSF@mathstodon.xyz · Greg Egan
- Crowdstrike have released their Technical Root Cause Analysis. In short - Checks didn't work as expected, validations didn't work as expected … Jon Robertson
- Crowdstrike External Technical Root Cause Analysis Lobsters
- Microsoft Accuses Delta of ‘False, Misleading’ Claims Over CrowdStrike Tech-Outage Barron's Online · Brian Swint
- Microsoft: Delta Struggled After IT Crash Because of Outdated Infrastructure PCMag · Kate Irwin
- Microsoft says Delta ignored Satya Nadella's offer of CrowdStrike help The Verge · Tom Warren
- Microsoft lashes out at Delta: Your ancient tech caused the service meltdown CNN · Ramishah Maruf
- Microsoft fires back at Delta after airline blames it for costly tech outage Associated Press
- Microsoft punches back at Delta Air Lines and its legal threats The Register · Jessica Lyons
- Microsoft hits back at Delta — says airline refused help following CrowdStrike outage, and even ignored an email from Satya Nadella himself TechRadar
- Microsoft responds to Delta's threats over $500 million CrowdStrike outage TweakTown · Jak Connor
- Microsoft joins Delta Air Lines pile on Cyber Daily · David Hollingworth
- CrowdStrike outage: Microsoft blames US airline for prolonged recovery impacting over 6,000 flights Business Today · Danny D'Cruze
- Microsoft Blames Delta's Outdated IT For Delayed Global Cyber Outage Recovery After Airline Decides To Seek Damages Against Software Giant And CrowdStrike Benzinga · Ananya Gairola
- Microsoft Says Outdated Tech Most Likely Caused Delta's Meltdown Skift · Meghna Maharishi
- ‘False, misleading and damaging’: Microsoft slams Delta for CrowdStrike outage blame Mashable · Alex Perry
- Microsoft joins CrowdStrike in pushing IT outage recovery responsibility back to Delta CIO Dive · Roberto Torres
- Microsoft fires back at Delta after massive outage, says airline declined ‘repeated’ offers for help NBC News
- CrowdStrike fights back against Delta Axios · Sam Sabin
- Microsoft snaps back at Delta, accuses airline CEO of sharing comments that were “incomplete, false, misleading, and damaging to Microsoft and its reputation” Windows Central · Sean Endicott
- Now Microsoft says Delta ignored its offers to help the airline with its CrowdStrike issues Neowin · John Callaham
- Microsoft blames Delta for its struggle to recover from global cyber outage Reuters
- Microsoft says Delta declined help for mass outage, calls for info on IBM and AWS cloud usage DatacenterDynamics
- Microsoft Hits Back at Delta in Clash Over System Breakdown Bloomberg · Mary Schlangenstein
- CrowdStrike: Amid Outage, Delta Turned Down Our Help Newser · Arden Dier
- Crowdstrike tells Delta threatened lawsuit will expose airline's own tech mis-steps The Stack · Joe Fay
- As I said, before about the CrowdStrike outages, I believe that with every modern enterprise PC having the capability to manage their fleet … Patrick Moorhead
Discussion
-
@iavins
@iavins
on x
Off by one strikes again [image]
-
@zoidctf
Zoid Kirsch
on x
Spoiler: It was regex. There are usually two things that break computers on the Internet: DNS and regex. This time it was regex.
-
@taliaringer
Talia Ringer
on x
Still think requiring that all third party kernel-space software use memory-safe languages is reasonable
-
@arekfurt
@arekfurt
on x
As l've said before regarding some other firms, I think CrowdStrike does deserve some credit for at least admitting in it postmortems how badly it screwed up technically. And boy, did it screw up technically: A sensor config file having one extra field was enough for a disaster. …
-
@rohan_devarc
@rohan_devarc
on x
“There is only one hard problem in computer science: off-by-1 error.” — Zig Steenine🙃 Even if Zig Steenine could write in Rust, it wouldn't have prevented this disaster. [image]
-
@grady_booch
Grady Booch
on x
Oops.
-
@crowdstrike
@crowdstrike
on x
This morning, we published the Root Cause Analysis (RCA) detailing the findings, mitigations and technical details of the July 19, 2024, Channel File 291 incident. We apologize unreservedly and will use the lessons learned from this incident to become more resilient and better
-
@uk_daniel_card
@uk_daniel_card
on x
“The CrowdStrike Falcon sensor delivers powerful on-sensor AI and machine learning...” did the AI forget to test as well??? /S
-
@tianyin_xu
Tianyin Xu
on x
Without effective configuration testing, embarrassing and disastrous failures like #CrowdStrike will be repeated again and again. > “CrowdStrike released a content configuration update... The sensor expected 20 input fields, while the update provided 21 input fields. The
-
@202accepted
@202accepted
on x
imagine if this happened during the election there's some timeline out there where that's the case [image]
-
@arekfurt
@arekfurt
on x
There are at minimum two or three separate ways this fact pattern represents astoundingly basic technical failure for a company that makes a product as important as CrowdStrike Falcon is.
-
@arekfurt
@arekfurt
on x
(For example: The update content verifier didn't even check that the number of fields the update had was no greater than the maximum it should have before the update went out? For real?)
-
@puponsecurity
@puponsecurity
on x
this is _not_ a useful root cause analysis. nothing addressed in it is the root of the issue. this was a procedural failure, not a technical one.
-
@royvanrijn
Roy van Rijn
on x
@CrowdStrike Basically: it was never tested in an environment which would be the same as released. Never was the combination providing 20 and needing 21 tested. Biggest lesson here: always test with real-world conditions; never assume inputs and use wildcards if the software itse…
-
@chrisalbon
Chris Albon
on x
Ooophf [image]
-
@erratarob
Robert Graham
on x
“We don't use signatures but use AI instead. Also, the bug was in a regex expression” — CrowdStrike
-
@bdsams
Brad Sams
on threads
I will die of irony if we find out the reason Delta was offline for so long was because of Lotus Notes
-
@david_slotnick
David Slotnick
on x
New: Microsoft says Delta declined multiple offers of help during last month's meltdown; accuses Delta of pushing narrative that's “incomplete, false, misleading, and damaging to Microsoft and its reputation.” Letter from Microsoft's attorney: [image]
-
@sysadafterdark
@sysadafterdark
on x
I've had a handful of Microsoft employees contact me regarding issues I've vocalized on here and they've given me beta builds and break/fix help at no cost. The support was stellar, and I'm surprised Delta turned down help only for them to turn around and bite their hand.
-
@andrewjmcclure
Andrew McClure
on x
CrowdStrike fires back at Delta While liability may be capped, not sure blaming your customers is a great long term strategy https://www.wsj.com/...
-
@tomwarren
Tom Warren
on x
Microsoft hits back at Delta, claiming that the airline ignored Satya Nadella's offer of free help for the CrowdStrike outage. Microsoft also suggests that Delta was actually struggling with non-Windows systems instead. Full details below 👇 https://www.theverge.com/...
-
@film_girl
Christina Warren
on x
My parents are fighting. https://www.theverge.com/... [image]
-
@kpottermn
Kyle Potter
on x
Not a good look for Delta. As its meltdown lingered, Microsoft says its CEO reached out directly to Delta CEO Ed Bastian on Wednesday, July 24. No response. By that point, Bastian was in Paris for the Olympics.
-
@paulrobichaux
@paulrobichaux
on x
Now Microsoft is clapping back at @Delta's claims about their sustained outage, including pointing out that MS offered free recovery help *and* that the scheduling system is hosted by IBM and isn't even running on Windows. https://www.theverge.com/...
-
r/technology
r
on reddit
Microsoft says Delta ignored Satya Nadella's offer of CrowdStrike help
-
r/delta
r
on reddit
Microsoft says Delta ignored Satya Nadella's offer of CrowdStrike help / Microsoft suggests that Delta Air Lines' issues were related to its old IT infrastructure — not Windows