Discussion

• @kylie.robison Kylie Robison on threads

  crowdstrike mood board ˚ᡣ𐭩 🧸ྀི｡ꪆৎ ˚ʚ🍓ɞ˚‧

• @crumbler Casey Newton on threads

  The CrowdStrike bug also doubles as an incredible unintentional art exhibit

• @joshuamarris Joshua Marris on threads

  It's still Microsoft's fault that one driver that doesn't even directly control hardware caused Windows to blue screen.  That's always been the case.

• @mckoss Mike Koss on threads

  I think Microsoft should have revoked crowdstrike's software certificates.  That way their flawed software that forced millions of computers into an infinite boot loop would not have even been installed to begin with.

• @paulkerton Paul Kerton on threads

  “Is it Ford's fault if you crash your car?” say people defending Microsoft's terrible decades of security decisions with Windows, reaching for nonsensical analogies.  If Ford let a third party directly access your car's engine software in order to provide a solution that wouldn't…

• @jordanwfrank Jordan Frank on threads

  Why the hell do all the computers that runs all the critical infrastructure on earth have that Counterstrike game on them in the first place.  That's what I want to know.  It seems unnecessary.

• @justjason24 Jason Young on threads

  The argument that CrowdStrike shouldn't be given kernel level access is getting out of hand.  How can a EDR effectively do its job if it's not running at the kernel level?  I know I'd prefer my trusted security suite looking at the memory of the entire device to effectively monit…

• @krishnanrohit Rohit on x

  What's the dollar value associated with Crowdstrike's screwup? And re the AI safety bills floating around, how would you prove this could not have been done with an LLM, if one was used anywhere in the chain, even for advice?

• @willguyatt Will Guyatt on x

  Lower than numbers that have been suggested elsewhere....

• @a_greenberg Andy Greenberg on x

  Microsoft now says the CrowdStrike crash hit 8.5 million Windows machines. https://blogs.microsoft.com/ ... I think that's the biggest disruption of computers ever. (Though maybe not the worst, given NotPetya and WannaCry did more lasting damage to hundreds of thousands of machin…

• @tomwarren Tom Warren on x

  Microsoft says the CrowdStrike faulty update has impacted 8.5 million Windows devices. That's a huge amount, but it's still less than 1% of all Windows machines so this could have been even worse if more devices had been hit 😬 https://blogs.microsoft.com/ ...

• @msftnews @msftnews on x

  Read an update on what we've done to help Microsoft customers recover from the recent CrowdStrike outage. Learn about our actions from the start of the incident and our collaboration with customers, cloud providers and others in the tech community. https://blogs.microsoft.com/ ..…

• @rakeshsfnyc Rakesh Agrawal on x

  TIL Blue Screen of Death did not originate with Windows, it came from OS/2. Raymond Chen at Microsoft is credited with bringing it to Windows 3.1. (Marketing must not have liked being associated with death.) It's still no Guru Meditation Error. @stevesi

• @thegrugq Thaddeus E. Grugq on x

  Not even a deci-Morris Worm!

• @snlyngaas Sean Lyngaas on x

  Microsoft on CrowdStrike-related outage: About 8.5 million Windows devices were affected by faulty Falcon update, or less than 1% of Windows machines on planet. https://blogs.microsoft.com/ ...

• @fxshaw Frank X. Shaw on x

  Helping our customers through the CrowdStrike outage https://blogs.microsoft.com/ ...

• @benweston88 Ben Weston on threads

  Also — directly on CrowdStrike's process: 1.  How the fuck did QA testing not pick up a bug with a 100% success rate at killing its target system?  2. Why the fuck don't a company with the userbase size and value they have operate a staged rollout policy?!  This isn't 2005. …

• @arcanedrifter @arcanedrifter on threads

  It can happen to anything.  Dont install unvetted and untested patched day 1 and during or before production hours.

• @weispaints Kevin Weis on threads

  I don't insist either way (despite gloating a bit in jest today lol) but I do know that it can be seen as an issue with how Windows allows 3rd party software access to the kernel.  It CAN happen on Mac or other Unix based systems but it's less likely due to their architecture. …

• @dino.luck Dino Luck on threads

  It is not a Windows issue.  It's like me putting a can of food in my fridge, that can explodes and breaks the fridge and I blame it on Samsung for making the fridge.

• @thekyefox Kye Fox on threads

  My understanding is Apple recently changed things around to block direct kernel access and manage stuff like this through an API.  It's not the perfect guard, but closes off a lot of possible problems.  Windows also provides APIs for it, but it still allows direct kernel access a…

• @technicallymims Christopher Mims on threads

  Interesting Hacker News thread on why today's outage could have happened to any OS running these critical systems (but I am not an expert and am ready to hear from anyone who insists this is also or even primarily a Windows issue) https://news.ycombinator.com/ ...

• @techronic9876 @techronic9876 on threads

  Apple's annoying sandboxing of system resources does have practical benefits

• @MaybeMyMonkeys@mastodon.social @MaybeMyMonkeys@mastodon.social on mastodon

  @Techmeme and didn't bother testing

• @JMarkOckerbloom@mastodon … John Mark Ockerbloom on mastodon

  Crowdstrike's statement on their Falcon content update includes the line: “We further recommend organizations ensure they're communicating with CrowdStrike representatives through official channels.”  It's a useful reminder that disruptions, besides causing problems directly …

• @prettyhatmachine.bsky.social @prettyhatmachine.bsky.social on bluesky

  Crowdstrike is just a symptom of the disease that is Microsoft.  —  No decent OS should be this vulnerable.

• @taviso Tavis Ormandy on x

  This strange tweet got >25k retweets. The author sounds confident, and he uses lots of hex and jargon. There are red flags though... like what's up with the DEI stuff, and who says “stack trace dump”? Let's take a closer look... 🧵1/n [image]

• @patrickwardle Patrick Wardle on x

  I don't do Windows but here are some (initial) details about why the CrowdStrike's CSAgent.sys crashed Faulting inst: mov r9d, [r8] R8: unmapped address ...taken from an array of pointers (held in RAX), index RDX (0x14 * 0x8) holds the invalid memory address @_JohnHammond [image]

• @eastdakota Matthew Prince on x

  @IAmDougLewis @CrowdStrike I guarantee you they have tight controls on code roll out. They have looser controls on config rollout. It's tough as a security company because you see a new threat and you want to fix it fast. You don't expect your own config to explode. But sometimes…

• @rakyll Jaana Dogan on x

  This gives insights why a typical staged rollout didn't catch the bug. CrowdStrike made a compromise to roll out config changes faster. In my experience, config changes are no different from code changes. And they are usually more error-prone than code. https://x.com/...

• @stevesi Steven Sinofsky on x

  Cause—coding error, testing oversight, specification incorrect, operator confusion, etc. Mechanism—divide by zero, pointer out of bounds, illegal operation, resource limit/contention, incorrect directions to operator, etc. Manner—software failed, hardware broke, networking

• @tomwarren Tom Warren on x

  this isn't the first time that CrowdStrike's csagent.sys kernel driver has caused Windows BSODs. I'd imagine many executives are waking up this morning and immediately looking at moving away from CrowdStrike. It's very hard to win back trust after an event like this

• @george_kurtz George Kurtz on x

  As CrowdStrike continues to work with customers and partners to resolve this incident, our team has written a technical overview of today's events. We will continue to update our findings as the investigation progresses. https://www.crowdstrike.com/ ...

• @stevesi Steven Sinofsky on x

  There needs to be a post outlining the manner, cause, and mechanism of the failure. Then the specific remediation. It feels like they are saying there was a corrupt descriptor file (mechanism = failure of format)—though these files are more than data and are likely a

• @jperlow @jperlow on x

  The beatings will continue until morale improves

• @norootcause @norootcause on x

  I gotta admit, named pipes is not something that comes up often in incident write-ups. Didn't even know that Windows supported them! https://www.crowdstrike.com/ ...

• @vkoukis Vangelis Koukis on x

  It's a shame that the technical bulletin on the global @CrowdStrike incident avoids being explicit about what the root cause was. So, let's embark on a bit of guessing. The bulletin, for context: https://www.crowdstrike.com/ ... [Thread ⬇️]

• @0xtib3rius @0xtib3rius on x

  Interesting line from the #CrowdStrike writeup: “This is not related to null bytes contained within Channel File 291 or any other Channel File.” (Channel Files are the .sys files which numerous people reported null bytes in) https://www.crowdstrike.com/ ...

• @tobycmurray Toby Murray on x

  ItCrowdStrike has since “clarified” ( https://www.crowdstrike.com/ ...): 1. It was not a “driver” but a (kernel loaded) “configuration file” that updated how Falcon “evaluated named pipe execution” 2. It was not related to null bytes (i.e. zeros) in the file Clear?

• @drandrewdwyer Andrew Dwyer on x

  Here's CrowdStrike's technical analysis... which says little about *how* or *why* this happened. I'm sure we'll find out in due course. https://www.crowdstrike.com/ ...

• @robmen Rob Mensching on x

  The technical details provided by Crowdstrike thus far refute some of the worst takes on Twitter. That's some goodness. Now we wait for the root cause analysis to answer the core question: Why wasn't this caught earlier (testing/staging/etc.)? Learning. https://www.crowdstrike.co…

• @gossithedog Kevin Beaumont on x

  Here's CrowdStrike's mini root cause analysis of what happened yesterday: https://www.crowdstrike.com/ ... It's basically exactly as commonly thought, i.e. a bad content update was pushed which caused the CrowdStrike driver to crash Bunch of clear learnings for CrowdStrike, e.g. …

• @techspence Spencer on x

  Ok so cs says despite the .sys it was not a kernel driver. I missed that part. Also calling it a logic error which makes it sound trivial. What am I missing? https://www.crowdstrike.com/ ...

• @jamiejbartlett Jamie Bartlett on x

  Criminals now looking to exploit this IT outage by claiming to be IT professionals ready to help. This is the most common trick in the book - as I wrote about here Be VERY wary of anyone turning up unannounced saying they'll help!

• @perpetualmaniac @perpetualmaniac on x

  Crowdstrike Analysis: It was a NULL pointer from the memory unsafe C++ language. Since I am a professional C++ programmer, let me decode this stack trace dump for you. [image]

• @loxyflo @loxyflo on x

  Anyone know how Liz Truss's first day at Microsoft is going?

• @jason @jason on x

  I guess crowdstrike doesn't do staged rollouts?

• @arekfurt @arekfurt on x

  If you haven't seen it, per Crowdstrike here's the concise explanation on how its bad updates actually wound up breaking Windows: (No more official technical detail at this time on what the “logic error” actually did at low-levels.) https://www.crowdstrike.com/ ... [image]

• @eastdakota Matthew Prince on x

  We should be careful creating incentives for systems' designers where when something goes wrong the right answer to satisfy the lawyers is to fail open. #thatsnotsecurity

• @shanselman Scott Hanselman on x

  Here's the thing folks. I've been coding 32 years. When something like this happens it's an organizational failure. Yes, some human wrote a bad line. Someone can “git blame” and point to a human and it's awful. But it's the testing, the Cl/CD, the A/B testing, the metered

• @stevesi Steven Sinofsky on x

  Kernel mode is *the* problem. In 2024 changing software from third parties via a private update channel is about the highest risk setup and should not be a generally available capability. And if it is it should not be used in critical systems.

• @k8em0 @k8em0 on x

  On the CrowdStrike outage: Most organizations of a certain size test software updates before deployment. They do not test “content updates” from OS or security software, but set them to automatically update because they are viewed as safe. IT departments just got a new daily task

• @hackerfantastic @hackerfantastic on x

  Are we *sure* the @CrowdStrike crash wasn't deliberate? They pushed a file full of NULL bytes to their agents which caused the BSoD...

• r/technology r on reddit

  What is CrowdStrike, and what happened?

• r/crowdstrike r on reddit

  Technical Details on Today's Outage

• @_driangle @_driangle on threads

  So Microsoft created an operating system that: 1.  Needs a 3rd party antivirus software, because they can't make it safe enough on its own. 2.  Can be totally bricked by an over the air update from a 3rd party. …

• @sbisson Simon Bisson on threads

  Remember when Microsoft tried to get AV vendors to use APIs rather than kernel drivers?  And they all said they couldn't and they were going to make it an antitrust issue?  Pepperidge Farm remembers.

• @johnnoonan John Noonan on threads

  Crowdstrike is at fault.  But so is Microsoft.  The architecture of the OS shouldn't allow a third party to just nuke Windows.

• @vthallam Venkatesh Thallam on threads

  One of the things I like about Meta is the blameless Sev(incident) reviews.  The senior engineers and leadership focuses on the systems that have lead to a Sev rather than focusing why an individual didn't foresee an edge case.  I've had Sev's that made it to Techcrunch front pag…

• @dco.st Demetrios on bluesky

  Like the Cavendish banana cultivar, which puts the world's banana supply at the potential mercy of a single pathogen, there are advantages to uniformity but the catastrophic potential is also high.  [embedded post]

• @linakhanftc Lina Khan on x

  1. All too often these days, a single glitch results in a system-wide outage, affecting industries from healthcare and airlines to banks and auto-dealers. Millions of people and businesses pay the price. These incidents reveal how concentration can create fragile systems.

• @jeffnolan Jeff Nolan on x

  @om @chrisfralic A glitch took down CDK is about the most egregious misstatement Lina Khan could make.

• @stevesi Steven Sinofsky on x

  This is concentrated *because* of regulation.

• @swiftonsecurity @swiftonsecurity on x

  Microsoft sales reps selling MDE while the Crowdstrike users are still down [video]

• @om @om on x

  The profound lack of understanding on how modern systems work, the legacy of decades old technology & reality is what I have come to expect from Washington. I will refrain from saying any more. And will leave a link to a piece I wrote here. https://om.co/...

• @allenholub @allenholub on x

  No amount of testing will guarantee perfection in a program. The real problem here is that Microsoft effectively allowed CrowdStrike to hack into the core of their operating system in the name of security. Maybe, they should pay less attention to AI nonsense and more attention to

• @sdw Sebastiaan de With on x

  @om absolutely drives me insane that all the regulators pushing the most impactful regulations seem to have the least understanding of what they are regulating

• @om @om on x

  @sdw All politics for sake of politics and personal political agenda greater than common sense, actual understanding and citizen interests. Jingoism is now life

• @stevesi Steven Sinofsky on x

  Arlines already use mobile devices for gate checkin, lounges, and kiosks. Hotels are the same. Even TSA. Hospitals already use connected systems via browser and/or Citrix. From now, the only strategy that is not negligence is to move critical infrastructure to mobile devices.

• @1612elphi Delphi on threads

  Microsoft allowing kernel extensions because of what's essentially lobbying from corporations is probably the most on brand thing I've heard from the enterprise IT crowd

• @fxshaw Frank X. Shaw on x

  This WSJ story is a great example of vibe-based journalism, where based on vibes the reporters decide what to write, then force-fit quotes into the narrative, blinking sheepishly when they end up in conflict. As in: “Friday's outage was caused by a buggy update sent to corporate

• @fxshaw Frank X. Shaw on x

  @yuusharo also, a Microsoft spokesperson would not have to make this point if the reporters did their jobs.

• @zeynep Zeynep Tufekci on x

  I think technical gloating goes to Apple — depreciated kernel extensions as of Big Sur — and Rust folks (who rarely need an excuse😬). The Crowdstrike debacle looks more Boeing, aka mismanagement and/or shortsighted cost cutting, no? Problem was their QA & update rollout steps.

• @swiftonsecurity @swiftonsecurity on x

  The correct answer is a fully validated and controlled execution environment needing no antivirus but we've made an industry of bandaids for a fundamental error in our approach.

• @stevesi Steven Sinofsky on x

  @SwiftOnSecurity Windows RT ;-)

• @fxshaw Frank X. Shaw on x

  @yuusharo from the article: A Microsoft spokesman said it cannot legally wall off its operating system in the same way Apple does because of an understanding it reached with the European Commission following a complaint. In 2009, Microsoft agreed it would give makers of security …

• @i0n1c Stefan Esser on x

  People pointing to EndpointSecurity framework in MacOS as the solution for the Crowdstrike problem are missing the point. ES is a typical Apple solution and basically means:anyone who can bypass it has to have exactly one exploit (chain) that will allow them to bypass ALL vendors

• @phil_stevens@mastodon.nz Phil Stevens on mastodon

  The best (so far) breakdown and attribution of responsibility/blame in the #crowdStrike fiasco that I've read so far.  Really lays a decent chunk of it at Microsoft's doorstep where it belongs (CS fucked up, bigtime, but MS didn't even vet the kernel driver code...WTAscreamingF) …

• @eastdakota Matthew Prince on x

  Here's the scary thing that's likely to happen based on the facts of the day if we don't pay attention. Microsoft, who competes with @CrowdStrike, will argue that they should lock all third-party security vendors out of their OS. “It's the only way we can be safe,” they'll

• @eastdakota Matthew Prince on x

  Everyone has a bad day. This one really sucked for @CrowdStrike. Continue to have faith in them as a partner and the best end point security solution on the market. #HugOps

• @dinodaizovi Dino A. Dai Zovi on x

  Good time to re-read “CyberInsecurity: The Cost of Monopoly” by Dan Geer et al: https://www.schneier.com/...

• @george_kurtz George Kurtz on x

  CrowdStrike CEO George Kurtz says “the issue has been identified, isolated and a fix has been deployed”, and CrowdStrike is working with its impacted customers