You're probably already planning this, but please setup an alarm to fire off if a new package release is published that is not correlated with a CI/CD run.

Or require manual intervention to publish a new package. I'm not sure why we need to have a fully automated pipeline here to go from CI/CD to public package release. It seems like having some kind of manual user interaction to push a new version of a library would be a good thing.

The basic issue with manual interaction is a question of authority: a pretty common problem for companies (and open source groups) is when $EARLY_EMPLOYEE/$EARLY_CONTRIBUTOR creates and owns the entire publishing process for a key package, and then leaves without performing a proper transfer of responsibility. This essentially locks the company/group out of its own work, and increases support load on community maintained indices to essentially adjudicate rightful ownership of the package name.

(There are a variety of ways to solve this, but the one I like best is automated publishing a la Trusted Publishing with environment mediated manual signoffs. GitHub and other CI/CD providers enable this.)

I don’t buy this. I mean, I’m sympathetic to the issue. It’s easy for things to get jumbled when companies are young. But we’re talking about libraries that are used by your customers. To many, this is the externally visible interface to the company.

What you describe sounds like a process problem to me. If an $EARLY_EMPLOYEE if the only one with the deploy keys for what is a product of the company, then that’s a problem. If a deployment of that key library can be made without anyone approving it, that’s also a problem. But those are both people problems… and you can’t solve a people problem with a technical solution.

> But those are both people problems… and you can’t solve a people problem with a technical solution.

I don’t think it is a people problem in this case: the only reason there’s a person involved at all is because we’ve decided to introduce one as an intermediating party. A more misuse-resistant scheme disintermediates the human, because the human was never actually a mandatory part of the scheme.

What person do you want to remove from the process?

The person who intermediates the trust relationship between the index and the source repository. There’s no reason for the credential that links those two parties to be intermediated by a human; they’re two machine services talking.

(You obviously can’t disintermediate the human from maintenance or development!)

But that’s the person I think is mandatory.

You’re saying that whatever is in the source repository should be uploaded in the npm index, right? If the code is tagged as release, the built artifact is automatically uploaded to npm. Is that what you’re proposing?

That exactly what got PostHog into this position. The keys to publish to npm were available to an engineer or GitHub to push a malware build into npm automatically. This isn’t a technical issue… it’s a process issue. I don’t see the problem as that the keys were misused. I see the problem as that it was possible to misuse the keys at all. Why do you need that process to be automatic? How often are you pushing new updates?

I would argue that those npm assets/libraries are your work product. That is what your customer needs to use your service. It is a published product from your company. It is too important to allow a new version to be published out to the public without a human in the loop to approve it.

When you have a fully automatic publishing cycle, you’re trading security for convenience. It’s all about how much risk you’re willing to accept. For me, that’s too much of a risk to the reputation to the company. I also think the balance shifts if you’re talking about purely internal assets, having completely automatic ci/cd makes perfect sense for most companies. For me, it is about who is hurt if there is an issue (and you should expect for there to be an issue).

Putting a person in the loop for releasing a product is one way to solve this. It’s not perfect, but at the moment, I think it’s the most secure (for the public).

> You’re saying that whatever is in the source repository should be uploaded in the npm index, right? If the code is tagged as release, the built artifact is automatically uploaded to npm. Is that what you’re proposing?

No, I'm saying that the source repository should act as an authentication principal itself. A human should still initiate the release process, but the authentication process that connects the source repository (more precisely CI/CD) to the index should not involve a credential that's implicitly bound to a human identity (because the human's role within a project or company is ephemeral).

As far as I can tell, what got PostHog into this situation wasn't a fully automated release process (very few companies/groups have fully automated processes), but the fact that they had user-created long-lived credentials that an attacker could store and weaponize at a time most convenient to them. That's a problem regardless of whether there's normally a human in the loop or not, because the long-lived credential itself was sufficient for publishing.

(In other words, we basically agree about human approval being good; what I'm saying is that we should formalize human approval without making the authentication scheme explicitly require an intermediating partner who doesn't inherently represent the actual principal, i.e. the source repository.)

I think we agree more than we don’t and the rest are personal preferences and policy differences. But we largely agree in principle.

I like the idea of having a person whose job is approving releases. Kind of like a QC tag — this release was approved by XX. I saw the issue as PostHog having a credential available to the CI/CD that had the authority to push releases automatically. When a new GitHub action was added, that credential was abused to push a bad update to npm. I might be wrong, I don’t deal with npm that much.

There are many ways to fix this.

You can't "require" manual intervention. Sure you can say that the keys stays on say 2 developers laptops, but personal devices have even more surface area for key leak than CI/CD pipeline. It wouldn't have prevented attacks like this issue in any case where the binary just searched for keys across the system.

One alternative is to do the signing on airlocked system stored in physically safe but accessible location, but I guess that's just way too much inconvenience.

As someone else mentioned, the easiest way would be to have some kind of MFA in the loop. It’s not perfect, but better than what we have now.

I get that it can be useful sometimes. But requiring physical MFA to make a package available to the general public seems like a no-brainer to me.

Users who really want to could opt in to the bleeding edge.

This is orthogonal to the issue at hand. The problem is a malicious actor cutting a release outside of the normal release process. It doesn't matter if the normal process is automated or manual.

It could have eliminated an attack surface where they steal the credentials from the CI/CD...

...But then you if I understand NPM publishing well, you would still have the credentials on someone's computer laying around? I guess you could always revoke the tokens after publishing? It's all balancing convenience and security, with some options being bad at both?

This is built in NPM. You can get an email on every pkg publishing.

Sure, it might be a little bit of noise, but if you get a notice @ 3am of an unexpected publishing, you can jump on unpublishing it.

Very nice way of putting it, kudos!

Did the client side JS being infected produce any issues which would have affected end users? As in if a web owner were on an affected version and deployed during the window would the end user of their site have had any negative impact?

No, just the host that was running the package (the exploit was pretty generic and not targeted at PostHog specifically). In fact, so far we think there were 0 production deployments of PostHog because the package was only live for a little bit.

Glad to hear the impact was so muted. Thank you for the response!

If we don't know how it got compromised, chances are this attack is still spreading?

If anything people should use an older version of the packages. Your newest versions had just been compromised, why should anyone believe this time and next time it will be different?!

The packages were published using a compromised key directly, not through our ci/cd. We rolled the key, and published a new clean version from our repo through our CI/CD: https://github.com/PostHog/posthog-js/actions/runs/196303581...

Why do you keep using token auth? This is unacceptable negligence these days.

NPM supports GitHub workflow OIDC and you can make that required, disabling all token access.

Yep, we are moving to workflow OIDC as the next step in recovery.

OIDC is not a silver bullet either and has its own set of vectors to consider too. If it works for your org model then great, but it doesn't solve every common scenario.

Trusted Publishing addresses the vector here, which is arbitrary persistence and delayed use of credentials by attackers. You're right that it's not a silver bullet (anything claiming to be one is almost certainly a financially induced lie), but it eliminates/foreshortens the attack staging window significantly.

It's not a silver bullet in that it can open up other vectors of much more probable attack.

If I have a FOSS project where there are other contributors with write access but not publish access (very common) OIDC now allows them to publish via workflows, opening you up to Jia Tan type attacks.

Like I said, if it works for your org, great. Otherwise it can cause bigger problems.

Sorry, what's the threat model in which you're giving people write access to your repository but assuming that they can't publish?

To establish a baseline here: the assumption with Trusted Publishing is that people are already using CI/CD to publish to places like PyPI. All Trusted Publishing does is replace the user-managed credential with a machine-intermediated one.

In other words: if someone had write access to your repository before, Trusted Publishing does not change their privilege, since they could always push a new CI/CD workflow that uses your manual publishing secret directly. All Trusted Publishing does is formalize the treatment of the source repository as a source of truth, and introduce some misuse-resistance properties (short-lived credentials, minimal self-scoping).

If this isn't your user model, that's perfectly fine. But in that case you shouldn't be using CI/CD to publish at all (which is also fine); it's not an issue with Trusted Publishing per se.

> Sorry, what's the threat model in which you're giving people write access to your repository but assuming that they can't publish?

So that I'm the arbiter of what people download, and can review locally what changes were made prior to pushing it out to everyone.

> To establish a baseline here: the assumption with Trusted Publishing is that people are already using CI/CD to publish to places like PyPI.

That's not the narrative. People have been saying to switch to trusted publishing from local publishes. They're not 1:1 compatible. I agree that if you already publish via CD, then it probably already worked for your org and thus is a no brainer.

> If this isn't your user model, that's perfectly fine. But in that case you shouldn't be using CI/CD to publish at all (which is also fine); it's not an issue with Trusted Publishing per se.

That's exactly the point I'm making.

As a user of Posthog, this statement is absurd: > Or safer again not to use software this vulnerable.

Nearly all software you use is susceptible to vulnerabilities, whether it's malicious or enterprise taking away your rights. It's in bad taste to make a comment about "not using software this vulnerable" when the issue was widespread in the ecosystem and the vendor is already being transparent about it. The alternative is you shame them into not sharing this information, and we're all worse for it.

Popularity and vulnerability go hand in hand though. You could be pretty safe by only using packages with zero stars on GitHub, but would you be happy or productive?

Glad you updated on this front-page post. Your Twitter post is buried on p3 for me right now. Good luck on the recovery and hopefully this helps someone.