/
Navigation
Chronicles
Browse all articles
Explore
Semantic exploration
Research
Entity momentum
Nexus
Correlations & relationships
Story Arc
Topic evolution
Drift Map
Semantic trajectory animation
Posts
Analysis & commentary
Pulse API
Tech news intelligence API
Browse
Entities
Companies, people, products, technologies
Domains
Browse by publication source
Handles
Browse by social media handle
Detection
Concept Search
Semantic similarity search
High Impact Stories
Top coverage by position
Sentiment Analysis
Positive/negative coverage
Anomaly Detection
Unusual coverage patterns
Analysis
Rivalry Report
Compare two entities head-to-head
Semantic Pivots
Narrative discontinuities
Crisis Response
Event recovery patterns
Connected
Search: /
Command: ⌘K
Embeddings: large
TEXXR

Chronicles

The story behind the story

days · browse · Enter similar · o open

Experts say CrowdStrike's fix requires deleting a specific file and that cannot be automated at scale, meaning outages could persist for longer than expected

and what the chaos serves to remind us about Katie Collins / CNET : Microsoft Outage: CrowdStrike Update Affects Flights, Hospitals and Businesses Globally Robert Greenall / BBC : Global services slowly recovering after bug causes IT chaos Business Insider : Mass IT outage: here's a list of companies and operations affected GovTech : Cybersecurity Update Causes Worldwide Microsoft Outages Trend Micro : Trend Experts Weigh in on Global IT Outage Caused by CrowdStrike Christianna Silva / Mashable : Which banks were affected by the Microsoft outage? What we know. Brian Krebs / Krebs on Security : Global Microsoft Meltdown Tied to Bad Crowdstrike Update NBC News : Global computer outage is one of the biggest in history Threads: Sung Kim / @sung.kim.mw : How to resolve this CrowdStrike issue.  This assumes you are also using BitLocker (kind of expected if you are using CrowdStrike).  Now do this for every impacted machines.  😀 Bluesky: @hammancheez.bsky.social : Crowdstrike : its fine u just have to manually visit the PC boot it into safe mode and remove a sys file  —  US Organization with 50,000 pcs and a completely outsourced IT department in Bangalore : what Jeremy Stanley / @jeremydstanley.com : seems like this is not true.  given enough reboots, machines will eventually download the fix — there's enough time for the machine to get online briefly before it enters the boot loop.  good job experts.  [embedded post] X: @vxunderground : How to fix the Crowdstrike thing: 1. Boot Windows into safe mode 2. Go to C:\Windows\System32\drivers\CrowdStrike 3. Delete C-00000291*.sys 4. Repeat for every host in your enterprise network including remote workers 5. If you're using BitLocker jump off a bridge @hammancheez : Crowdstrike : its fine u just have to manually visit the PC boot it into safe mode and remove a sys file US Organization with 50,000 pcs and a completely outsourced IT department in Bangalore : what Tom Warren / @tomwarren : it's amazing that the CrowdStrike fix is literally “ have you tried turning it off and on again?” It's working for some IT admins! [image] @swiftonsecurity : You could build a PXE boot WIM file and have it execute a fix script but that will require telling everyone how to boot over the network. Very few have this skillset though and will likely require reconfiguring every network to do DHCP relay and won't work if machine locked down. @swiftonsecurity : You will also need the local admin LAPS password to do this... And many machines have a broken WinRE environment at least on the disk. Yeah it's pretty grim recovery situation in theory for any moderately complex organization... Dylan Patel / @dylan522p : Y2K24 - fuckload Windows Machines are absolutely fucked. Crowdstrike $crdw down 19% premarket They pushed out a buggy update (.sys files are kernel drivers, Crowdstrike's agent lives in the kernel) People's computers crashed It also fucks up booting into loading All affected @swiftonsecurity : Note this will not work if your machine is bitlocker encrypted without getting the recovery key for each machine... Theo / @t3dotgg : Pouring one out for all the IT people who have to explain “safe mode” to Carl on the sales team

New York Times

Context & Ripple Effects

The immediate workaround was to boot affected Windows machines in Safe Mode and remove the faulty driver file, but BitLocker recovery-key requirements could obstruct that path on encrypted endpoints. The operational bottleneck sits alongside the broader risk that endpoint-security products have privileged access to the operating system core, as related coverage noted in the endpoint-security tools' OS-level access.

Later coverage attributed the crashes to a Falcon sensor configuration update that produced a logic error and was remediated within 78 minutes. That short vendor-side remediation did not eliminate the customer-side recovery burden of manually removing the affected driver from each machine.

First-order effects

  • Affected organizations must allocate IT staff to recover endpoints individually, extending disruption to operations dependent on those machines, including businesses, hospitals, and flights.
  • BitLocker-protected systems can require recovery keys before remediation, turning endpoint encryption controls into an immediate constraint on restoration workflows.

Second-order effects

  • Customers will scrutinize endpoint-security vendors' update controls and recovery procedures, because a rapidly withdrawn update can still leave a long tail of devices needing hands-on repair.
  • IT teams may need to reassess whether their device-management, key-escrow, and incident-response processes can recover machines when the normal operating environment is unavailable.

Third-order effects

  • The incident highlights distribution-layer liability: software that operates at the OS core concentrates both security value and outage risk, increasing pressure for safer release gates and rollback designs.
  • If comparable failures recur, buyers are likely to treat recoverability under endpoint failure—not only detection performance—as a core criterion for security-platform selection.

The trend: Security software is becoming critical operational infrastructure, making resilient update delivery and device recovery as consequential as the protection the software provides.

Discussion

  • @sung.kim.mw Sung Kim on threads
    How to resolve this CrowdStrike issue.  This assumes you are also using BitLocker (kind of expected if you are using CrowdStrike).  Now do this for every impacted machines.  😀
  • @hammancheez.bsky.social @hammancheez.bsky.social on bluesky
    Crowdstrike : its fine u just have to manually visit the PC boot it into safe mode and remove a sys file  —  US Organization with 50,000 pcs and a completely outsourced IT department in Bangalore : what
  • @jeremydstanley.com Jeremy Stanley on bluesky
    seems like this is not true.  given enough reboots, machines will eventually download the fix — there's enough time for the machine to get online briefly before it enters the boot loop.  good job experts.  [embedded post]
  • @hammancheez @hammancheez on x
    Crowdstrike : its fine u just have to manually visit the PC boot it into safe mode and remove a sys file US Organization with 50,000 pcs and a completely outsourced IT department in Bangalore : what
  • @tomwarren Tom Warren on x
    it's amazing that the CrowdStrike fix is literally “ have you tried turning it off and on again?” It's working for some IT admins! [image]
  • @swiftonsecurity @swiftonsecurity on x
    You could build a PXE boot WIM file and have it execute a fix script but that will require telling everyone how to boot over the network. Very few have this skillset though and will likely require reconfiguring every network to do DHCP relay and won't work if machine locked down.
  • @swiftonsecurity @swiftonsecurity on x
    You will also need the local admin LAPS password to do this... And many machines have a broken WinRE environment at least on the disk. Yeah it's pretty grim recovery situation in theory for any moderately complex organization...
  • @dylan522p Dylan Patel on x
    Y2K24 - fuckload Windows Machines are absolutely fucked. Crowdstrike $crdw down 19% premarket They pushed out a buggy update (.sys files are kernel drivers, Crowdstrike's agent lives in the kernel) People's computers crashed It also fucks up booting into loading All affected
  • @swiftonsecurity @swiftonsecurity on x
    Note this will not work if your machine is bitlocker encrypted without getting the recovery key for each machine...
  • @t3dotgg Theo on x
    Pouring one out for all the IT people who have to explain “safe mode” to Carl on the sales team