What we know about the xz Utils backdoor that almost infected the world
What we know about the xz Utils backdoor that almost infected the world
Malicious updates made to a ubiquitous tool were a few weeks away from going mainstream.
Specifically, SSH logins were consuming too many CPU cycles and were generating errors with valgrind, a utility for monitoring computer memory.
Through sheer luck and Freund’s careful eye, he eventually discovered the problems were the result of updates that had been made to xz Utils. On Friday, Freund took to the Open Source Security List to disclose the updates were the result of someone intentionally planting a backdoor in the compression software.
It is lucky that Andres Freund checked and found the issue in valgrind that was maybe intentionally or maybe unintentionally.
https://www.openwall.com/lists/oss-security/2024/03/29/4
I'm interested in figuring out what happened and more information on the contributor behind the attack.
56Everybody keeps saying it was found by luck, but this seems like it was found by a guy who maintains his repository properly and monitors his CPU cycles diligently just to prevent this sort of thing. I guess I would call it lucky he found it so quickly, but it was definitely not found by luck.
64The fact that that person happened to be looking on a system downstream to this one, while also having the context needed to pin it back to xz in particular is the lucky part. The same attack in any of countless other places wouldn't have gotten spotted the same way, or as quickly. That's not to say diligence on Freund's part wasn't a big factor here, but it's important to identify that luck was a big factor.
32It's more so lucky that there was someone diligently doing that. It could've easily gone unnoticed had there not been someone like him.
20Yep. It could also be found by luck a bit later, or caught in a strict process sooner or later that we won't know.
No process is perfect, but the fact this attack is super complex, with many levels of indirection on run time, detecting if it's in debug mode, forcing maintainers to disable Oss fuzzer (a tool that scans source code for bugs and vulnerabilities) and also involved social engineering and took years to get close to widespread release is to show what it takes to plant a back door on 100% Foss systems.
As opposed to closed source, where all it takes is a conversation between govt and a CEO.
16
to me: the most chilling thing is that someone involved in the open source ecosystem introduced this vulnerability and, if it was intentional, what else did they do?
11It isn't if it was intentional. It was intentional. Otherwise the exploit chain wouldn't be so convoluted.
9
At the moment, it’s unknown if there was ever a real-world person behind this username or if Jia Tan is a completely fabricated individual.
Well, I'm pretty sure it wasn't a platypus. There was definitely one or more real-world people behind this. I doubt anyone thinks that their name is actually Jia Tan, though.
41I saw someone trying attribute this to Hong Kong based on the possible origins of the name.
Seems like a stretch to me
5Failing that hard to not know what is, and isn't knowable, is a good reason to not give a single f about what they think
4
Incredible, just incredible. I am looking forward to the upcoming time when we know more about exactly how it works.
21Yeah it sounds pretty wild already with some kind of, like, door knock mechanism using certificates? So you can't scan for it. And some reverse engineering countermeasures.
Like everyone else, I have to wonder what libraries have been compromised in a way that nobody has noticed yet.
15From what I've read is not authentication bypass, it's a RCE using certificates to deliver the payload. If a specific signature is found it runs the code that was sent in place of the signing public key. It also means that only someone who has the ability to generate that specific key signature could use the RCE.
There were some other bits that looked like they could have been placed to enable compromising other build systems in the future when they checked for xz support.
2
It's crazy how they pressured/manipulated the maintainer. Especially fucked up considering he wasn't in a good mental state and was still helping the community by maintaining FOSS software.
8This is the best summary I could come up with:
They were likely very close to seeing the backdoor update merged into Debian and Red Hat, the two biggest distributions of Linux, when an eagle-eyed software developer spotted something fishy.
Andres Freund, a developer and engineer working on Microsoft’s PostgreSQL offerings, was recently troubleshooting performance problems a Debian system was experiencing with SSH, the most widely used protocol for remotely logging in to devices over the Internet.
Through a combination of sheer luck and Freund’s careful eye, he eventually discovered the problems were the result of updates that had been made to xz Utils.
On Friday, Freund took to the Open Source Security List to disclose the updates were the result of someone intentionally planting a backdoor in the compression software.
Malicious code added to xz Utils versions 5.6.0 and 5.6.1 modified the way the software functions when performing operations related to lzma compression or decompression.
For instance, Tan replaced Collins' contact information with their own on oss-fuzz, a project that scans open source software for vulnerabilities that can be exploited.
The original article contains 640 words, the summary contains 171 words. Saved 73%. I'm a bot and I'm open source!
8So if I don't use SSH am I fine? Because my distro doesn't let me remove the package. Other packages depend on it.
4Unless you're running Debian testing you're safe. If ssh isn't open to the internet you're safe. Just make sure everything is up-to-date.
7Debian testing has 'updated' to
5.6.1+really5.4.5-1anyway, so as long as you've updated within the past few days it will have been downgraded to 5.4.5.8
You can't remove it but you can downgrade.
2
How quickly was it caught?
2After a month from what I read, most distro hadn't included it yet
Edit: it had been prepped for 2 years or so and the malicious code was introduced a while ago, but from release it took about a month to detection
6