/
Navigation
Chronicles
Browse all articles
Explore
Semantic exploration
Research
Entity momentum
Nexus
Correlations & relationships
Story Arc
Topic evolution
Drift Map
Semantic trajectory animation
Posts
Analysis & commentary
Pulse API
Tech news intelligence API
Browse
Entities
Companies, people, products, technologies
Domains
Browse by publication source
Handles
Browse by social media handle
Detection
Concept Search
Semantic similarity search
High Impact Stories
Top coverage by position
Sentiment Analysis
Positive/negative coverage
Anomaly Detection
Unusual coverage patterns
Analysis
Rivalry Report
Compare two entities head-to-head
Semantic Pivots
Narrative discontinuities
Crisis Response
Event recovery patterns
Connected
Search: /
Command: ⌘K
Embeddings: large
TEXXR

Chronicles

The story behind the story

days · browse · Enter similar · o open

Reeling from CrowdStrike-related outages, Delta has canceled 5,000+ flights, including 1,384 on Sunday and 700+ for Monday so far, according to FlightAware

Delta Air Lines (DAL.N) struggled to restore normal operations on Sunday after last week's crippling global cyber outage …

Reuters

Discussion

  • @carnage4life Dare Obasanjo on threads
    I've seen people giving the CEO of CrowdStrike a hard time for not apologizing in his social media posts about they took down many of their customers.  Delta alone had to cancel 4,500 flights.  I expect lawsuits to follow and CrowdStrike doesn't want to accept blame in writing.
  • @thechrisflyer Chris Dong on x
    Experiencing a @Delta meltdown firsthand. Flight canceled, self-service rebooking tool broken on both web and app, and long lines at Logan. This is days after the Crowdstrike outage and no other carrier is having issues this badly.
  • @airlineflyer Jason Rabinowitz on x
    As with many other airline tech outages before #CrowdStrike, it wasn't the outage itself that sunk the airline. It was Delta's own systems being unable to cope with the high number of cancellations and track its crews to staff flights once the initial outage was resolved. [image]
  • @deltanewshub @deltanewshub on x
    An update to Delta customers from CEO Ed Bastian https://news.delta.com/...
  • @krisvancleave Kris Van Cleave on x
    After another day w/ 1000+ cancelations stemming from the CrowdStrike IT outage, Delta is hoping to turn the corner and operate a more normal day tomorrow...while finding itself in unfamiliar territory—the DOT hotseat. This from @SecretaryPete after speaking with Delta's CEO: [im…
  • @carnage4life Dare Obasanjo on x
    At least one company isn't going to be renewing their CrowdStrike subscription.
  • @secretarypete Secretary Pete Buttigieg on x
    We have received reports of continued disruptions and unacceptable customer service conditions at Delta Air Lines, including hundreds of complaints filed with @USDOT. I have made clear to Delta that we will hold them to all applicable passenger protections.
  • @robleathern Rob Leathern on x
    @Carnage4Life I doubt this matters that much - not apologizing is not going to stop class action lawyers not the resulting discovery
  • @1612elphi Delphi on threads
    Microsoft allowing kernel extensions because of what's essentially lobbying from corporations is probably the most on brand thing I've heard from the enterprise IT crowd
  • @ianb@mastodon.well.com Ian Betteridge on mastodon
    The 2009 deal, by the way, was nothing to do with security software: it was the final settlement of the long-running antitrust case which focused on browser choice and interoperability.  The proposals on interop - which are probably the ones MS is referring to here - were made by…
  • @fxshaw Frank X. Shaw on x
    @yuusharo also, a Microsoft spokesperson would not have to make this point if the reporters did their jobs.
  • @r00tkitsmm Meysam on x
    in Intel based macOS, Crowdstrike's falcon utilizes a KEXT which could potentially be vulnerable. however on Apple Silicon it employs System Extensions instead. thanks Apple for keeping kernel a forbidden place :)
  • @malwarejake Jake Williams on x
    Please, oh person who has never done any kernel development and thinks that anything with a .sys extension must be a kernel driver, bless me with your hot takes on how modern security products should be architected...
  • @counternotions Kontra on x
    ‘Searches for “Microsoft outage” outranked “CrowdStrike outage” on Google consistently from Friday morning through Saturday morning.’ https://www.wsj.com/... Apple News: https://www.wsj.com/...
  • @gossithedog Kevin Beaumont on x
    Re the 'it's Microsoft fault for letting EDR drivers do dumb shit' argument - there are safety mitigations MS made but many EPP vendors work around them (including CS) - MS tried to make the space much safer back about 15 years ago, but vendors like Mcafee pushed against it
  • @andrewmayne Andrew Mayne on x
    This is insane. According to WSJ: Microsoft says the European Commission made it illegal for them to block the level of access that made the Crowdstrike outage possible. [image]
  • @fxshaw Frank X. Shaw on x
    from the article: A Microsoft spokesman said it cannot legally wall off its operating system in the same way Apple does because of an understanding it reached with the European Commission following a complaint.  In 2009, Microsoft agreed it would give makers of security software …
  • @fxshaw Frank X. Shaw on x
    This WSJ story is a great example of vibe-based journalism, where based on vibes the reporters decide what to write, then force-fit quotes into the narrative, blinking sheepishly when they end up in conflict. As in: “Friday's outage was caused by a buggy update sent to corporate
  • @stevesi Steven Sinofsky on x
    @SwiftOnSecurity Windows RT ;-)
  • @swiftonsecurity @swiftonsecurity on x
    The correct answer is a fully validated and controlled execution environment needing no antivirus but we've made an industry of bandaids for a fundamental error in our approach.
  • @i0n1c Stefan Esser on x
    People pointing to EndpointSecurity framework in MacOS as the solution for the Crowdstrike problem are missing the point. ES is a typical Apple solution and basically means:anyone who can bypass it has to have exactly one exploit (chain) that will allow them to bypass ALL vendors
  • @zeynep Zeynep Tufekci on x
    I think technical gloating goes to Apple — depreciated kernel extensions as of Big Sur — and Rust folks (who rarely need an excuse😬). The Crowdstrike debacle looks more Boeing, aka mismanagement and/or shortsighted cost cutting, no? Problem was their QA & update rollout steps.
  • @eastdakota Matthew Prince on x
    Here's the scary thing that's likely to happen based on the facts of the day if we don't pay attention. Microsoft, who competes with @CrowdStrike, will argue that they should lock all third-party security vendors out of their OS. “It's the only way we can be safe,” they'll
  • @benweston88 Ben Weston on threads
    Also — directly on CrowdStrike's process: 1.  How the fuck did QA testing not pick up a bug with a 100% success rate at killing its target system?  2. Why the fuck don't a company with the userbase size and value they have operate a staged rollout policy?!  This isn't 2005. …
  • @arcanedrifter @arcanedrifter on threads
    It can happen to anything.  Dont install unvetted and untested patched day 1 and during or before production hours.
  • @weispaints Kevin Weis on threads
    I don't insist either way (despite gloating a bit in jest today lol) but I do know that it can be seen as an issue with how Windows allows 3rd party software access to the kernel.  It CAN happen on Mac or other Unix based systems but it's less likely due to their architecture. …
  • @dino.luck Dino Luck on threads
    It is not a Windows issue.  It's like me putting a can of food in my fridge, that can explodes and breaks the fridge and I blame it on Samsung for making the fridge.
  • @thekyefox Kye Fox on threads
    My understanding is Apple recently changed things around to block direct kernel access and manage stuff like this through an API.  It's not the perfect guard, but closes off a lot of possible problems.  Windows also provides APIs for it, but it still allows direct kernel access a…
  • @technicallymims Christopher Mims on threads
    Interesting Hacker News thread on why today's outage could have happened to any OS running these critical systems (but I am not an expert and am ready to hear from anyone who insists this is also or even primarily a Windows issue) https://news.ycombinator.com/ ...
  • @techronic9876 @techronic9876 on threads
    Apple's annoying sandboxing of system resources does have practical benefits
  • @MaybeMyMonkeys@mastodon.social @MaybeMyMonkeys@mastodon.social on mastodon
    @Techmeme and didn't bother testing
  • @prettyhatmachine.bsky.social @prettyhatmachine.bsky.social on bluesky
    Crowdstrike is just a symptom of the disease that is Microsoft.  —  No decent OS should be this vulnerable.
  • @taviso Tavis Ormandy on x
    This strange tweet got >25k retweets. The author sounds confident, and he uses lots of hex and jargon. There are red flags though... like what's up with the DEI stuff, and who says “stack trace dump”? Let's take a closer look... 🧵1/n [image]
  • @rakeshsfnyc Rakesh Agrawal on x
    Absurd that Crowdstrike is claiming they had a resolution in one hour. While that *may* be technically true, having left customers computers in a state where they couldn't install the fix makes it realistically untrue.
  • @firstadopter Tae Kim on x
    CrowdStrike deserves the blame. They failed basic testing QA, which is unacceptable at their customer scale and kernel access. No one has been more critical (and correct) about Microsoft's poor gaming practices and strategy than me, but Microsoft isn't at fault here. [image]
  • @patrickwardle Patrick Wardle on x
    I don't do Windows but here are some (initial) details about why the CrowdStrike's CSAgent.sys crashed Faulting inst: mov r9d, [r8] R8: unmapped address ...taken from an array of pointers (held in RAX), index RDX (0x14 * 0x8) holds the invalid memory address @_JohnHammond [image]
  • @stevesi Steven Sinofsky on x
    Cause—coding error, testing oversight, specification incorrect, operator confusion, etc. Mechanism—divide by zero, pointer out of bounds, illegal operation, resource limit/contention, incorrect directions to operator, etc. Manner—software failed, hardware broke, networking
  • @rakyll Jaana Dogan on x
    This gives insights why a typical staged rollout didn't catch the bug. CrowdStrike made a compromise to roll out config changes faster. In my experience, config changes are no different from code changes. And they are usually more error-prone than code. https://x.com/...
  • @eastdakota Matthew Prince on x
    @IAmDougLewis @CrowdStrike I guarantee you they have tight controls on code roll out. They have looser controls on config rollout. It's tough as a security company because you see a new threat and you want to fix it fast. You don't expect your own config to explode. But sometimes…
  • @perpetualmaniac @perpetualmaniac on x
    Crowdstrike Analysis: It was a NULL pointer from the memory unsafe C++ language. Since I am a professional C++ programmer, let me decode this stack trace dump for you. [image]
  • @tomwarren Tom Warren on x
    this isn't the first time that CrowdStrike's csagent.sys kernel driver has caused Windows BSODs. I'd imagine many executives are waking up this morning and immediately looking at moving away from CrowdStrike. It's very hard to win back trust after an event like this
  • @stevesi Steven Sinofsky on x
    There needs to be a post outlining the manner, cause, and mechanism of the failure. Then the specific remediation. It feels like they are saying there was a corrupt descriptor file (mechanism = failure of format)—though these files are more than data and are likely a
  • @george_kurtz George Kurtz on x
    As CrowdStrike continues to work with customers and partners to resolve this incident, our team has written a technical overview of today's events. We will continue to update our findings as the investigation progresses. https://www.crowdstrike.com/ ...
  • @jperlow @jperlow on x
    The beatings will continue until morale improves
  • @fxshaw Frank X. Shaw on x
    Helping our customers through the CrowdStrike outage https://blogs.microsoft.com/ ...
  • @norootcause @norootcause on x
    I gotta admit, named pipes is not something that comes up often in incident write-ups. Didn't even know that Windows supported them! https://www.crowdstrike.com/ ...
  • @vkoukis Vangelis Koukis on x
    It's a shame that the technical bulletin on the global @CrowdStrike incident avoids being explicit about what the root cause was. So, let's embark on a bit of guessing. The bulletin, for context: https://www.crowdstrike.com/ ... [Thread ⬇️]
  • @0xtib3rius @0xtib3rius on x
    Interesting line from the #CrowdStrike writeup: “This is not related to null bytes contained within Channel File 291 or any other Channel File.” (Channel Files are the .sys files which numerous people reported null bytes in) https://www.crowdstrike.com/ ...
  • @tobycmurray Toby Murray on x
    ItCrowdStrike has since “clarified” ( https://www.crowdstrike.com/ ...): 1. It was not a “driver” but a (kernel loaded) “configuration file” that updated how Falcon “evaluated named pipe execution” 2. It was not related to null bytes (i.e. zeros) in the file Clear?
  • @drandrewdwyer Andrew Dwyer on x
    Here's CrowdStrike's technical analysis... which says little about *how* or *why* this happened. I'm sure we'll find out in due course. https://www.crowdstrike.com/ ...
  • @robmen Rob Mensching on x
    The technical details provided by Crowdstrike thus far refute some of the worst takes on Twitter. That's some goodness. Now we wait for the root cause analysis to answer the core question: Why wasn't this caught earlier (testing/staging/etc.)? Learning. https://www.crowdstrike.co…
  • @gossithedog Kevin Beaumont on x
    Here's CrowdStrike's mini root cause analysis of what happened yesterday: https://www.crowdstrike.com/ ... It's basically exactly as commonly thought, i.e. a bad content update was pushed which caused the CrowdStrike driver to crash Bunch of clear learnings for CrowdStrike, e.g. …
  • @techspence Spencer on x
    Ok so cs says despite the .sys it was not a kernel driver. I missed that part. Also calling it a logic error which makes it sound trivial. What am I missing? https://www.crowdstrike.com/ ...
  • @jamiejbartlett Jamie Bartlett on x
    Criminals now looking to exploit this IT outage by claiming to be IT professionals ready to help. This is the most common trick in the book - as I wrote about here Be VERY wary of anyone turning up unannounced saying they'll help!
  • @loxyflo @loxyflo on x
    Anyone know how Liz Truss's first day at Microsoft is going?
  • @jason @jason on x
    I guess crowdstrike doesn't do staged rollouts?
  • @arekfurt @arekfurt on x
    If you haven't seen it, per Crowdstrike here's the concise explanation on how its bad updates actually wound up breaking Windows: (No more official technical detail at this time on what the “logic error” actually did at low-levels.) https://www.crowdstrike.com/ ... [image]
  • @eastdakota Matthew Prince on x
    We should be careful creating incentives for systems' designers where when something goes wrong the right answer to satisfy the lawyers is to fail open. #thatsnotsecurity
  • @shanselman Scott Hanselman on x
    Here's the thing folks. I've been coding 32 years. When something like this happens it's an organizational failure. Yes, some human wrote a bad line. Someone can “git blame” and point to a human and it's awful. But it's the testing, the Cl/CD, the A/B testing, the metered
  • @stevesi Steven Sinofsky on x
    Kernel mode is *the* problem. In 2024 changing software from third parties via a private update channel is about the highest risk setup and should not be a generally available capability. And if it is it should not be used in critical systems.
  • @k8em0 @k8em0 on x
    On the CrowdStrike outage: Most organizations of a certain size test software updates before deployment. They do not test “content updates” from OS or security software, but set them to automatically update because they are viewed as safe. IT departments just got a new daily task
  • @hackerfantastic @hackerfantastic on x
    Are we *sure* the @CrowdStrike crash wasn't deliberate? They pushed a file full of NULL bytes to their agents which caused the BSoD...
  • r/technews r on reddit
    CrowdStrike's Falcon Sensor also linked to Linux kernel panics and crashes
  • r/technology r on reddit
    What is CrowdStrike, and what happened?
  • @_driangle @_driangle on threads
    So Microsoft created an operating system that: 1.  Needs a 3rd party antivirus software, because they can't make it safe enough on its own. 2.  Can be totally bricked by an over the air update from a 3rd party. …
  • @sbisson Simon Bisson on threads
    Remember when Microsoft tried to get AV vendors to use APIs rather than kernel drivers?  And they all said they couldn't and they were going to make it an antitrust issue?  Pepperidge Farm remembers.
  • @johnnoonan John Noonan on threads
    Crowdstrike is at fault.  But so is Microsoft.  The architecture of the OS shouldn't allow a third party to just nuke Windows.
  • @vthallam Venkatesh Thallam on threads
    One of the things I like about Meta is the blameless Sev(incident) reviews.  The senior engineers and leadership focuses on the systems that have lead to a Sev rather than focusing why an individual didn't foresee an edge case.  I've had Sev's that made it to Techcrunch front pag…
  • @ianb@mastodon.well.com Ian Betteridge on mastodon
    The scary bit of this is that CrowdStrike only affected 1% of all Windows PCs, and yet had such an impact.  What would a real cyberattack look like - one that, say, took out 5% of PCs?  —  https://blogs.microsoft.com/ ...
  • @dco.st Demetrios on bluesky
    Like the Cavendish banana cultivar, which puts the world's banana supply at the potential mercy of a single pathogen, there are advantages to uniformity but the catastrophic potential is also high.  [embedded post]
  • @linakhanftc Lina Khan on x
    1. All too often these days, a single glitch results in a system-wide outage, affecting industries from healthcare and airlines to banks and auto-dealers. Millions of people and businesses pay the price. These incidents reveal how concentration can create fragile systems.
  • @stevesi Steven Sinofsky on x
    This is concentrated *because* of regulation.
  • @jeffnolan Jeff Nolan on x
    @om @chrisfralic A glitch took down CDK is about the most egregious misstatement Lina Khan could make.
  • @sdw Sebastiaan de With on x
    @om absolutely drives me insane that all the regulators pushing the most impactful regulations seem to have the least understanding of what they are regulating
  • @om @om on x
    @sdw All politics for sake of politics and personal political agenda greater than common sense, actual understanding and citizen interests. Jingoism is now life
  • @om @om on x
    The profound lack of understanding on how modern systems work, the legacy of decades old technology & reality is what I have come to expect from Washington. I will refrain from saying any more. And will leave a link to a piece I wrote here. https://om.co/...
  • @swiftonsecurity @swiftonsecurity on x
    Microsoft sales reps selling MDE while the Crowdstrike users are still down [video]
  • @allenholub @allenholub on x
    No amount of testing will guarantee perfection in a program. The real problem here is that Microsoft effectively allowed CrowdStrike to hack into the core of their operating system in the name of security. Maybe, they should pay less attention to AI nonsense and more attention to
  • @stevesi Steven Sinofsky on x
    Arlines already use mobile devices for gate checkin, lounges, and kiosks. Hotels are the same. Even TSA. Hospitals already use connected systems via browser and/or Citrix. From now, the only strategy that is not negligence is to move critical infrastructure to mobile devices.