Microsoft engineer Andres Freund accidentally found the malicious code in versions of the XZ Utils compression tool, likely preventing thousands of infections

Yesterday's discovery of the xz backdoor was an accident. But what a fortunate accident it was.

Security Boulevard 2024-04-01 Mike Larkin

Context & Ripple Effects

The incident followed reports that compromised XZ Utils releases had reached Linux distributions including Debian and Red Hat, creating a potential remote-code-execution path. The subsequent attack timeline indicates this was a long-running effort against a widely used open-source component, not a routine software defect.

Freund's detection matters because it interrupted the chain before the affected releases could become a broader operational problem. Later warnings from open-source security groups that the attempt may not be isolated put scrutiny on how critical volunteer-maintained dependencies are reviewed and funded.

First-order effects

Linux distributors and downstream users of the affected XZ Utils versions must identify, remove, and validate compromised packages; maintainers face an immediate code-review and release-integrity response.
The discovery limits immediate exposure from the backdoor, while giving security teams indicators and a concrete case for investigating related build and contributor activity.

Second-order effects

Distribution vendors and enterprise Linux users are pushed to audit transitive dependencies and package provenance, rather than treating repository inclusion as sufficient assurance.
Maintainers of other foundational projects face pressure to strengthen review coverage and escalation paths, especially where a small number of contributors oversee broadly deployed code.

Third-order effects

If similar incidents emerge, open-source ecosystems may shift toward more formal support, independent review, and reproducible-build practices for critical dependencies rather than relying primarily on maintainer trust.
The case reinforces that software supply-chain defense must cover contributor influence and release processes as well as vulnerability scanning; how broadly such controls are adopted remains uncertain.

The trend: The XZ incident is part of a broader shift from securing individual applications to securing the trust relationships and maintenance pipelines behind widely reused open-source components.

Discussion

@AndresFreundTec@mastodon.social AndresFreundTec on mastodon
I was doing some micro-benchmarking at the time, needed to quiesce the system to reduce noise. Saw sshd processes were using a surprising amount of CPU, despite immediately failing because of wrong usernames etc. Profiled sshd, showing lots of cpu time in liblzma, with perf una…
@mhoye@mastodon.social @mhoye@mastodon.social on mastodon
Sad prediction: as long as disaster is averted by the heroic efforts of unpaid volunteers, people who measure success in dollars will consider the system to be working correctly as is, so because it was caught before it could do catastrophic damage by volunteer effort, the xv rev…
@AndresFreundTec@mastodon.social AndresFreundTec on mastodon
One more aspect that I think emphasizes the number of coincidences that had to come together to find this: — I run a number “buildfarm” instances for automatic testing of postgres. Among them with valgrind. For some other test instance I had used -fno-omit-frame-pointer for s…
@AndresFreundTec@mastodon.social AndresFreundTec on mastodon
There are more coincidences that are even less interesting. But even the above should make it clear how unlikely it was that I found this thing.
@AndresFreundTec@mastodon.social AndresFreundTec on mastodon
Afaict valgrind would not have complained about the payload without -fno-omit-frame-pointer. It was because _get_cpuid() expected the stack frame to look a certain way. — Additionally, I chose to use debian unstable to find possible portability problems earlier. Without that …
@AndresFreundTec@mastodon.social AndresFreundTec on mastodon
I accidentally found a security issue while benchmarking postgres changes. — If you run debian testing, unstable or some other more “bleeding edge” distribution, I strongly recommend upgrading ASAP. — https://www.openwall.com/...
@filippo.abyssdomain.expert Filippo Valsorda on bluesky
Yeah ok this was 100% found by sheer luck. mastodon.social/@AndresFreun... bsky.app/profile/look... [embedded post]
@shenetworks @shenetworks on x
Every time something like this happens I wonder what I'm even doing in this field 😅 because what do you mean you're micro-benchmarking your SSH connection
@vxunderground @vxunderground on x
The xz backdoor was initially caught by a software engineer at Microsoft. He noticed 500ms lag and thought something was suspicious. This is the Silver Back Gorilla of nerds. The internet final boss. [image]
Checkmarx.com Checkmarx.com on x
Backdoor Discovered in xz: The Most Advanced Supply Chain Attack Known to Date
@alex.stamos Alex Stamos on threads
This back door is the most interesting hack of the year and proves what everybody suspected about the supply-chain risks of OSS. Should hopefully drive some serious investment by the companies that profit from open-source to look for back doors using scalable means. https://boeh…
@jasonkoebler Jason Koebler on threads
The Xz backdoor highlights all of the good things about open source projects and also all of the vulnerabilities and weirdness associated with relying on essentially random volunteers to build software that huge parts of the internet use https://www.404media.co/...
@postmastron John Mastron on threads
The xz supply chain compromise is nuts. - It's hidden as a test case for a corrupt archive. - The injection is only in the source tar and not the git repo. - The backdoor only targets enterprise Linux distros that patch sshd to notify systemd, eg RHEL/Debian. …
@djlink@mastodon.gamedev.place David Amador on mastodon
A good example why caring about software performance does matter. The xz backdoor could have stayed hidden longer if not for someone thinking “heck, this seems slow”. We have multiples of faster hardware, yet software is lagging. — https://mastodon.social/... [image]
@swiftonsecurity @swiftonsecurity on x
Maybe I'm just being a spoilsport here but someone should point out it's not your job to worry about an attack like xz unless you are in a very precise role and this is fun but you should get back to disabling NTLMv1 on Monday and patching your network devices.
@prerationalist @prerationalist on x
babe whats wrong your 500ms slower than usual
@ollieatnowhere Ollie Whitehouse on x
On the xz ‘event’ this is the best ongoing technical analysis seen so far of the final .o payload https://gist.github.com/...
@andresfreundtec Andres Freund on x
@binitamshah FWIW, I didn't actually start looking due to the 500ms - I started looking when I saw failing ssh logins (by the usual automated attempts trying random user/password combinations) using a substantial amount of CPU. Only after that I noticed the slower logins.
@thegrugq @thegrugq on x
The problem isn't maintainer burnout. That was just the vulnerability that was exploited this time. Intelligence agencies aren't waiting for the right opportunity to come around, they find or create opportunities when they want them. Stop blaming mental health.
@natfriedman Nat Friedman on x
Worrying about every 500ms of unexpected latency now
@pinboard @pinboard on x
The idea that a Microsoft engineer would notice latency is easily the least credible part of this whole story.
@tobi Tobi Lutke on x
Engineers like this are rare and deserve to be celebrated. Most of what we all do every day only works due to their great skill at their craft
@amasad Amjad Masad on x
You know that annoying person on your team who insists every bit of perf regression needs to be investigated. One day they will save the world.
@matthew_d_green Matthew Green on x
So apparently the xz backdoor used an Ed448 signature to sign a RCE payload. I'm fascinated by the decision to toss away 48 bytes of payload space on a > 128-bit secure signature. Someone is being very smart or pretty dumb.
@rauchg Guillermo Rauch on x
A great skill to hone as an engineer: knowing how fast something should be from first principles. 500ms in the abstract seems fast. Blink of an eye ish. But added to a rejected SSH session... not so much.
@seldo Laurie Voss on x
I think the immediate lesson of the xz backdoor is that we got very lucky finding it, which probably means the other 100 times they tried we were not so lucky.
@mscccc Mike Coutermarsh on x
MSFT stock grants New hire: can't code. has 30k twitter followers. $800k over 4 years. Current eng: finds 500ms backdoor, saves world. $67k retention grant, vests over 5 years.
@vxunderground @vxunderground on x
Microsoft engineer: 500ms lag in liblzma? Something's up. Also Microsoft engineer: 45 minute lag in Microsoft Teams? Perfect.
@eastdakota Matthew Prince on x
Trying to come up with analogy to explain #xzbackdoor to my mom. Best I've come up with so far: Imagine a grizzled, tired, old oil pipeline maintainer was approached by a plucky young kid who was new to the small town the oil pipeline passed through. Slowly, over years, the kid..…
@satyanadella Satya Nadella on x
Love seeing how @AndresFreundTec, with his curiosity and craftsmanship, was able to help us all. Security is a team sport, and this is the culture we need everywhere. https://securityboulevard.com/ ...
@fr0gger_ Thomas Roccia on x
🤯 The level of sophistication of the XZ attack is very impressive! I tried to make sense of the analysis in a single page (which was quite complicated)! I hope it helps to make sense of the information out there. Please treat the information “as is” while the analysis... [image]
@thegrugq @thegrugq on x
If the xz maintainer wasn't burnt out then the agency would find another susceptibility, maybe he needs money, or has relationship problems, or likes to drink or gamble or collect rare books, or something. Anything. Intelligence agencies do their homework and play for keeps.
@thegrugq @thegrugq on x
The xz backdoor was the final part of a campaign that spanned two years of operations. These operations were predominantly HUMINT style agent operations. There was an approach that lasted months before the Jia Tan persona was well positioned to be given a trusted role.
@tylerglaiel Tyler Glaiel on x
the wildest part of this story is someone at microsoft actually thinking 500ms is bad and not just going “eh, its fine”
@_saagarjha Saagar Jha on x
The security community is going through the five stages of grief right now with the xz thing and I think a lot of people are coping with “there are technical measures that could have prevented this”. To move on, it is important to understand that this is not true in the slightest
@julianor Juliano Rizzo on x
Jia Tan's git commit to turn off Landlock sandboxing one week after Lasse Collin improved it. I understand the sandbox is for xz, the command line tool, and Jia did not need to disable it for the SSHD backdoor. 🤔The xz command also activates the backdoor? [image]
@bakedsec Austin Baker on x
“The nation-state supply chain plot was foiled, not by the NSA or FBI, but Steve from the build team when his test build process took 837ms instead of 214ms like it always had.” It's Mission Impossible, but with a firmware engineer to hunt down the upstream latency root cause.
r/ProgrammerHumor r on reddit
anyOneFollowingOnXzBackdoorCase

Chronicles