Points for Collin for letting his holiday take precedence over this mess.
Don’t forget that he made xz for free, as a hobby project, and likely got duped by “Jia Tan” same as everybody else did. He’s not obligated to solve this on any particular timeline.
Xz is not a business, so if your business got in trouble because a single solo hobby dev was a bit too trustful, it’s your job, and not his, to mitigate the problem.
I understand what you are saying and for open source community to have a good reputation and keep its standards high, these important packages have to start being actually maintained by the enterprises that depend on these. Most of businesses use Linux for their servers mainly Ubuntu/Debian and its just crazy for them to be vulnerable in such ways. Imagine how many other places backdoors can be installed even right now, and many of them won’t be visible and wont have performance issues like this one. 0.5s is a big deal in software and if it was 50ms or 5ms no one would have known this possibly for years. I wonder how safe the software we use actually is.
These open source projects have a good reputation and high standards because the developers freely volunteering their time care about the project. Forcing these developers to give their projects to Big Corp may not be the big improvement you are imagining.
Software may have backdoors, I'm not convinced this couldn't have happened if a large company was managing the product. AFAICT, open source software has a better security track record, in general.
IMHO, if it is anyone's job to look for and prevent these kinds of backdoors, it's probably companies like RedHat and Canonical. They bundle the tools with their products and they charge money for support.
Backdoor detection is an unsolvable problem both in theory and in practice. We are talking about carefully hidden obfuscated backdoors in MLOC codebases, which will have unexplained binaries and arcane build scripts. No company, not even Google, can guarantee that they'll be able to catch every backdoor hidden in some arbitrary codebase.
The only practical solution is for these companies to rewrite the core packages from scratch, written from the ground up so that they are easy to audit - no unexplained binaries, no arcane build scripts, etc.
> AFAICT, open source software has a better security track record, in general.
iirc Studies was done on this and there was no measurable difference in security issues when it comes to open vs closed source.
Just because the code is visible doesn't mean much, as you need to have the right eyes to actually notice security issues and a lot of open source projects don't have this.
Also closed source projects have the advantage where if they have a massive company behind them, that company has the resources (i.e. $$$) to hire highly specialised people to look the security of the software. Open source projects usually don't have such resources, heartbleed and xz are indicative of that.
Yes and people are paying top money to Canonical and RedHat to maintain these software don’t you think? They have huge enterprise contracts and I feel like this should be on them? Im not blaming anyone it just goes to show how vulnerable they are. Also other linux distros, Fedora, arch, etc. they have the money and should make their software safe don’t you think?
Fedora is supported by Red Hat. And yes, I do think there is something to the fact that Red Hat is selling businesses commercial support for projects they have no expertise in, which probably means some of the pressure flows downwards from Red Hat's clients to the unpaid maintainers.
Arch, (Gentoo, Nix, Mint, etc.) have no commercial aspect on the other hand and certainly doesn't have the money, but those distros are offered on the same "as is" basis as the downstream software.
> Yes and people are paying top money to Canonical and RedHat to maintain these software don’t you think? They have huge enterprise contracts
Money alone makes poor filler material to patch such holes. Microsoft famously doesn't make their stuff for free, and yet somehow Windows systems have gained this reputation of having holes such that a train can go through.
People build trust, take shortcuts, cut corners all the time. It's how we are wired. Doing otherwise is an energy sink for our bodies. We don't make ourselves overspend energy when we can.
Software isn't tooth brushes. The simpler and more specialized it is the cheaper it is to maintain. The more complex it is the harder it is to maintain.
Software R&D is a fixed cost. An ERP package with its at most 10,000 installations amortizes that cost on far fewer users than Linux with its millions of licensed installs.
First, the problem is the reputation hit and the trust going away. Right now on Hacker News 2 maybe 3 of the posts on front page have been about this since the news broke out. Also when trust goes away, its very hard to come back.
Secondly, the main problem that I see is how many other backdoors/dependencies are vulnerable that we might not know? They might not have performance issues. Also if this went unnoticed, in the long term it might have found its way and actually compromised people. I’m glad this was taken care of and no one was compromised.
OpenSSH is developed by the OpenBSD project and I have a lot more confidence in them than in any random "enterprise".
The issue in this case is that Linux distros took it upon themselves to alter its code base by linking in libsystemd (thus also liblzma) for the dubious benefit of better systemd integration, which comes with a generous helping of attack surface.
> Also other linux distros, Fedora, arch, etc. they have the money and should make their software safe don’t you think?
Nope. With open source the user is responsible. It's been this way from day one. Come to think of it, almost every piece of software I've ever used that is commercial has a no warranties and not suited for purpose clause in the EULA... because all the components the vendor used have that clause in the license to the vendor.
There's no connection between how much money somebody charges for their services and the quality of the services that they in turn provide.
Much of the worst professional service I've ever received, despite paying them a lot of money, has been from the example you gave, doctors.
Doctors aren't alone, of course. The most expensive restaurants I've ever been to have been some of the worst dining experiences I've ever had, while friends or relatives inviting me over for a backyard barbecue have been among the best, for example.
I've also found this to be the case for lawyers, accountants, teachers/professors, mechanics, and other well-compensated "professionals".
Software hasn't been an exception, either. I've had more success with free software than I have had with the equivalent paid offerings many times over.
In general, I've found that service quality is far more tied to the provider's level of passion for the task at hand, rather than anything to do with the amount of financial compensation that may be involved.
The software we use is not safe at all. If we don't have an idea what goes on in an open source projects, how could we even begin to know what goes on in a proprietary binary, or a service? People's dealings operate on trust, that's how, and trust can and will be abused, simple as that.
>Most of businesses use Linux for their servers mainly Ubuntu/Debian
Businesses should calculate with the risks, and also, use Long Term Support versions of software, not unstable bleeding edge fresh from the oven ones. Which they do. And so they were not vulnerable at all.
It's not the maintenance of the projects the enterprises have to fund (though that is certainly welcome if the mantainers are willing) but the slow, painful work of vetting the code.
Google does that with Project Zero but few companies are wealthy enough to afford that. The way out is economic, not technical: insurance, and mutualizing the cost of security audits. I wrote up my ideas on the subject here:
What are you actually proposing here? Any Ubuntu/Debian distribution contains hundreds or thousands of packages and transitive dependencies - it's unreasonable to expect corporations to contribute, even financially, to all of these.
This is supposed to be what services such as Tidelift provide - companies pay them, and OSS maintainers sign up with them for funding (and other support? unsure). But it's tough to apply this through the whole OSS ecosystem. To me, this should be what Canonical's support contracts should be for. They're perfectly positioned to help mediate these types of issues.
Im proposing that high security packages like this that effect ssh and main parts of the OS critical for security and safety should be maintained by actual enterprises not by one developer. Im not saying they should maintain everything, only the high importance parts of it. They should make sure that one bad actor cant compromise the whole ecosystem. I really don’t think what im saying is that far out of reach or that crazy.
Edit: also, why should a package like ssh have so many random dependencies? They should make sure these kind of packages have the least of dependencies especially on these kind of packages that have one maintainer to reduce the security risks?
why should a package like ssh have so many random dependencies
It's worth noting that vanilla upstream OpenSSH DOESN'T depend on xz. The dependency was patched in by the major distros to better integrate with systemd - and it's (AIUI) a transitive dependency from systemd - even the patch doesn't use it, but it gets pulled in with other systemd stuff.
that's what i believed as well, but i didn't take time to verify. and i didn't know about the details. thanks. as you describe it ssh still doesn't depend on xz (and why would it?) so part of the problem here is software architecture.
how is it possible that a seemingly unrelated dependency somewhere within systemd can affect and be exploited through ssh directly?
shouldn't it be possible to keep that separate?
doesn't openssh itself already implement some form of privilege separation?
how does software architecture here and in general need to change to prevent things like this?
i am sure somewhere these questions are already being discussed. i'd appreciate any pointers.
why should a package like ssh have so many random dependencies
that's the critical question here. it's not that enterprises need to take over maintenance of xz, but that the critical packages and their dependencies need to be audited on a regular basis. the development of xz is ok. what it needs is help with code reviews. and if it can't receive those, then it needs to be removed as a dependency of openssh.
Yes, exactly, these high security packages that every distro uses must be very tightly maintained and looked over with the least dependencies. SSH is one of the most important packages in security and it just feels clumsy that they are using such packages like XZ. Just WHY? Who made that decision? Why doesn’t all the distros work together maybe to maintain the security? Why depend on one person that does this as a hobby?
> high security packages like this that effect ssh and main parts of the OS critical for security and safety should be maintained by actual enterprises not by one developer.
I don’t remember the company names, but I guess people will remember the incidents.
- Solarwinds have been breached end to end.
- A company has been bribed (forced?) to ship backdoored encryption algorithms.
- A network hardware supplier’s firmware had been backdoored by Chinese IIRC.
- NSA backdoored national standards.
- Microsoft has been breached end to end.
In short, even if you’re a company, you’re one NSL, one bad actor, one misstep away from “total pwnage”.
I trust some individuals for developing critical software than entire “enterprise”s.
Actual enterprises do not have a better track record. I would go as far to say that if Big Corp maintained an SSH client and server then a lot of distributions would shun it out of distrust. How many products from actual enterprises phone home with telemetry or leak data in other ways?
> Edit: also, why should a package like ssh have so many random dependencies? They should make sure these kind of packages have the least of dependencies especially on these kind of packages that have one maintainer to reduce the security risks?
Using standard libraries for common stuff like compression, cryptography and whatnot is vastly more preferable over everyone shipping their own crypto, or worse, patches of crypto (see the Debian SSH key vulnerability of 2008 for an example [1]). For protocols it's in the end just as bad, it's a nightmare to keep different versions of the same program to be able to talk to each other, but now imagine a literal ton of programs who all have a wild mixture of statically shipped libraries, homegrown stuff that has barely been tested... no, just no. Not a world I'd like to live in.
> What are you actually proposing here? Any Ubuntu/Debian distribution contains hundreds or thousands of packages and transitive dependencies - it's unreasonable to expect corporations to contribute, even financially, to all of these.
Why is it unreasonable?
I actually see it as very reasonable. You use a package in a commercial distribution, you use aggregators like Github sponsors to pay the maintainer (a subscription, not a one-off payment!). What's unreasonable about that?
I don't really see how paying the maintainers helps against insider threats. The users still has to ensure that the software is safe.
Paying the maintainers is nice, but I think the megacorps and governments has to fund an organization that does security audits of all the sensitive packages, instead of heaping yet another job on the shoulders of the maintainers. It's especially important to not saddle the developers of open source with red tape, most of them would think that getting paid is poor compensation for the lost freedom.
See the rest of my comment, where I propose using an aggregator. I think it's unreasonable to expect everyone to engage in support contracts individually with every maintainer.
Proprietary Unix vendors used to develop all the system libraries and utilities. It's very much doable, especially if the responsibility is split across Linux vendors. Vendors need to take a critical look at what they're using, who develops it etc.
Well, I'm certain Facebook would be glad if xz is replaced everywhere by zstd which they maintain. You can't force them to pay for xz when they already maintain a replacement.
I've said before, elsewhere, that I think that that cartoon is misleading. The truth, as should be apparent nowadays after the parade of leftpad, ntpd, xz, et al., is that there are thousands of those little blocks at the bottom that have a bus factor of 1 volunteer. There aren't any big long base blocks at all. And sometimes the little blocks seem as if they have been set up by domino topplers. (-:
But for the comics' physical analogy to work you'd then need to draw it in thousands of dimensions so that any block breaking would bring down the edifice...
Linux Kernel. libc. Compilers. Programming Languages. Big libraries like filesystems, while generally not open source, system ROMs. Tons of network and system RFCs, standards like POSIX.
These are these long, wide, thin slabs you don't see. Others are too high level. These slabs are very close to metal. You're not kneeling enough. :)
Compilers and programming languages tend to be under supported (or overcomitted?) for what it's worth. They're definitely not solo projects, but certain features and components can be basically that way.
I take some of the recent concern from the Rust community about burnout and such to be early stage discovery of the economic problems in maintaining nontrivial infrastructure as a common good.
I'm sorry but no. If you are responsible for the worst backdoor in recent computing history (which is probably a criminal act in nature), then you do have a responsibility to explain what went down. Maybe he can take a couple days to write it but I think more than that would be irresponsible. Collin is partially responsible because he gave the keys to Jia Tan. I don't want to burn him at the stake since he's a victim too but I think it's reasonable to expect him to help clean up this mess and at least help us fill in the gaps and understand what happened.
Think about it this way. Say you are volunteering for a non-profit. You obviously don't have to do it and could just chill at home instead. But once you have agreed to say take a volunteer shift on Saturday, you are obligated to show up. Obviously no one can really stop you from skipping, but it would look bad on you and may get you banned from the non-profit. Along the same token, you obviously shouldn't steal from the non-profit, or say randomly beat up the customers/clients. If you accidentally set the building on fire which ends up killing 5 people, I would also assume you would help in some fashion to clean up.
Just because you are doing things for free does not mean you don't have social responsibility for your actions. You don't have to work, but just saying "oh my software ends up compromising everyone's computer? sorry not my problem" is seriously misunderstanding your role (I'm not saying that Collin is doing that). Similarly, if I just randomly slapped you in the face, I might have done it for free, but you would also probably be pretty pissed at me and expect an apology.
Just want to point out that this isn't a typical "demand underpaid open source developers to work 24/7" issue here. This is a unique circumstance where his action has directly resulted in this happening. I feel bad for him being guilt-tripped to add a maintainer after reading through the old threads two years ago, and as I said I don't want to burn him at the stake, just saying that saying he has no responsibility to help at all is really a bad take.
Some important context here is that Lasse Collin posted the LKML message and the web page on Saturday the 30th, so while the topmost comment argues that he had no obligation to respond promptly, he nevertheless did respond promptly.
> has no responsibility to help at all is really a bad take.
He has no responsibility to help at all.
All of the work done is that of a donation. Everyone who used xz did so for their benefit. Collin may feel a personal duty to work in the interests of the users of xz, but that's their exclusive choice. Any claim otherwise is on par with saying required-donation.
> "oh my software ends up compromising everyone's computer? sorry not my problem" is seriously misunderstanding your role (I'm not saying that Collin is doing that).
Then why mention it at all? But even if you did want to say that, that's true! You don't blame the victim of some abuse. Everyone, you included (assuming you use xz in anyway) were lied to. Just because Collin was lied to first, and most, doesn't change the fact the person who did all of the important work was also lied to and betrayed.
> Similarly, if I just randomly slapped you in the face, I might have done it for free, but you would also probably be pretty pissed at me and expect an apology.
Personally I'd be neither pissed, nor would I expect an apology. Shitty people are gonna be shit. But your mistake is blaming Collin for 'introducing' you to the person that slapped you in the face. But you're arguing as if Collin should have known this guy was a shitbag, and should hold the icepack while you berate them for making the mistake of trying to help other people... wtf?!
This us vs ourselves toxicity needs to stop. It's not us vs us. It's not even us vs the them. It's us vs the problem, us vs the malicious. The person writing the code is the good guy here.
> ...non-profit... no one can really stop you from skipping, but it would look bad on you
No it won't. Unless you have obligations.
> Along the same token, you obviously shouldn't steal from the non-profit, or say randomly beat up the customers/clients.
That's unrelated, to say the least. You obviously shouldn't steal from anyone nor beat up anyone, that's not related to this specific case at all.
> Just because you are doing things for free does not mean you don't have social responsibility for your actions.
No. No one forced the society to use the work that somebody was doing for free. And if society decided that it's fine to use it, it's on the society to take the associated risks. Everything else is a good will of the maintainer.
The stealing part is related because in this case, his project is "stealing" (or injecting a backdoor"). Does that make the analogy clearer?
To be exact, he didn't do it specifically, but the non-profit is usually locked but he unlocked the door to let someone in and that person ended up stealing from the non-profit.
I really don't understand why people do not see that if you cause a giant mess, you have some responsibility to clean up after yourself. Again, I'm not saying that they should be obligated to do work. But this isn't "working". This is cleaning up an issue that you have caused.
> No. No one forced the society to use the work that somebody was doing for free. And if society decided that it's fine to use it, it's on the society to take the associated risks. Everything else is a good will of the maintainer.
If I voluntarily baked a cake for you for free because I was nice and I mistakenly poisoned you, would you say "Oh i decided to take the risk of eating your cake anyway, not your fault"?
I didn't have to bake the cake. But since I decided to, I have basic obligations including not putting poison in it.
You are confusing legal and societal obligations, again. OpenSource maintainers and coders don’t owe you anything, so please stop pretending that they do.
I'm not confused about it. I never said Collin is legally liable to help. I'm talking about societal obligations here. Just because you do something for free doesn't mean your actions don't have consequences.
Yes you are, when talking about the poisoned cake. Poisoning it is a crime.
> because you do something for free doesn't mean your actions don't have consequences.
Of course there are consequences - society is a closed system. Though it doesn’t give you ANY right to demand anything from the person who you didn’t pay to, directly or indirectly. Stop juggling the facts and whining, no one owes you anything, as I already said.
I won't say it's the worst one (assuming the attacker's private key is not public), just one of the most interesting ones in regards to the social engineering, pressure applied, code hidden in the test files etc.
I would say it's definitely the most interesting one we know about so far :)
Is it because thousands of companies and millions of users depend on it?
On one guy on holiday working for free!
When we are essentially in two cold wars currently with ICBM nuclear powers?
Linux is the battleground of cyber warfare, which is being funded by nations probably in the hundreds of billions of dollars. A trillion dollars or more of economic activity runs on it
Windows? Come on, even buggier and less secure, and far less transparent.
Monetary support of Linux maintainers is now a national security concern. You can't have hundred billions for attack and zero dollars for defense, that is a stupid strategy.
Lasse has been active and answering questions on IRC over the last day or so. He says he'll be sharing more information on the XZ site in the days or weeks to come.
He's doing fine, by the way, and mentions that the messages of support are appreciated but not necessary.
He's more focused now on figuring out what happened, how he missed it, and deciding a plan of action for cleaning things up.
(paraphrasing from conversations in the public channels)
No, I was going for questioning and help with their investigations.
Seems like something serious and illegal happened. Possibly to do with a hostile state. Where I live, there would be at least 2 different organisations that would be very interested to find out what happened as quickly as possible. I don't for one second think he was involved but he would be essential in any investigation.
If you look at the list of (former) contributors of Tukaani, you will notice that an overwhelming majority of the names are Finnish, Lasse included, though he could also pass for Swedish. So FBI is going to investigate Github at most, unless Lasse has suddenly moved to the US or FBI has invaded Finland in some capacity.
All meaningful new information will probably come from identifying Jia Tan, if that ever happens. I haven't done much reading yet, but it seems to me there was no connection between him and Lasse outside the Github project.
the discussion here is very different from previous discussions because it focuses on the role of the maintainer and not the incident itself. so no, we haven't already discussed this.
We sure do, and this is a good thing. Some even here don't take care of themselves enough (including, but not limited to, taking proper breaks and other time for themselves).
Yes, that is kind of the point. The internet is full of bullies and that isn’t going to change - his heart wasn’t in it and he had bad judgement. He was played, I get it, but I know a tonne of professions where you’d be out the door if you messed up on that scale.
You’re not making any sense, he “had bad judgement” in getting an other maintainer so he should hand out maintainership? Like… to who?
> I know a tonne of professions where you’d be out the door if you messed up on that scale.
Cool story bro, know what happens in “a tonne of professions”? You get paid for doing stuff, and you get colleagues, with an actual formal legal contract, who also get paid.
And yet there are still bad actors, flipping insiders or getting moles in is literally standard modus operandi in espionage.
The guy made it obvious he was struggling. He needed somebody to take it over and someone came to his ‘rescue’… and he trusted every single change they brought.
Why am I saying that you don’t make sense? …because you haven’t even suggested a single resolution. You attack me even though I’m saying it how it is.
Now how about making use of your reply and suggesting something more useful? Because the status quo is actually what led to this, people just ignoring the issue and leaving the guy to his devices when his heart clearly isn’t in it.
Can I remind everyone that the entire world was nearly hacked?
You're very confident that this could have been avoided so can you point out EXACTLY what he should have done differently?
As far as I can tell, the only way to detect the manipulated release would have been to compare the release tarballs with one built by independently to detect the extra build script that had been inserted that injected the backdoor.
Of course, anybody in the world could have done this, including you, and yet... Neither you, nor anybody else, did. The reality here is that the entire supply chain is at fault - source code should be built from the public repo, not from a tarball.
Many very experienced people, and public companies, allowed this to happen. It's not fair to pick on one person.
Yes, Lesse, there are many people who rely on this project and it is sad to see it fall into its current state. You must hand it over to someone who knows what they are doing. Now!
Don’t forget that he made xz for free, as a hobby project, and likely got duped by “Jia Tan” same as everybody else did. He’s not obligated to solve this on any particular timeline.
Xz is not a business, so if your business got in trouble because a single solo hobby dev was a bit too trustful, it’s your job, and not his, to mitigate the problem.