Please be careful when revoking tokens. It looks like the payload installs a dead-man's switch at ~/.local/bin/gh-token-monitor.sh as a systemd user service (Linux) / LaunchAgent com.user.gh-token-monitor(macOS). It polls api.github.com/user with the stolen token every 60s, and if the token is revoked (HTTP 40x), it runs rm -rf ~/.
Yes, that would be one potential solution. But I have certainly never done it and bet >99.999% of the world's use of sudo is through 'sudo'.
Plus you only need one slip-up and you're hosed. Even people who try to almost always use '/usr/bin/sudo' will undoubtedly accidentally let a 'sudo' go through. Maybe they copy/paste a command from somewhere (after verifying that it's safe of course) and just didn't think of the sudo issue then and there.
The real problem is that there should be at least 2 levels for sudo, one for installing software and another that really allows someone to compromise the entire system, both layers should be separate to mitigate risk. At least the most secure layer should allow you to perform secure recovering and diagnosis
If you set the appropriate linux capabilities flag on a binary such as sshd at bootup then unprivileged users can bind to 22, no problem.
setcap 'cap_net_bind_service=+ep' /usr/sbin/sshd
Could even run it as a daemon unprivileged from a home directory with "systemd --user"
That said if you have multiple users and want every user to have their own sshd reachable on port 22 on the same machine you probably want to listen on vhost namespaced unix sockets and have something like haproxy listen on port 22 instead. Haproxy could of course also run unprivileged provided it has read access to all the sockets.
For that you really only need CAP_NET_BIND_SERVICE.
The bigger issue is that if you want to install or update system-wide packages, many of those will be used by privileged processes. Suppose you want to update /bin/sh. Even if the only permission you had is to write binaries, that'll get you root.
The old bin user is an idea that could be modernized with a new two level sudo concept, the higher one for recovery and diagnosis, already done in Chromebook and other solutions
bin passwords I will always remember: At the University of Maryland CS department systems the bin password was "fuck,you", and there was a devout Christian student on staff who had a problem with that, so we had to change it (to something harder to remember, I just can't recall).
Ok, so the malware runs a keylogger / clipboard logger, gets the password and runs sudo on it's own. Or replaces your shell by putting exec ~/hackedbash into your bashrc
Password on sudo is only useful if you detect the infection before you run sudo
Physical attestations are hard to solve, I think it would be nice if all TPMs in laptops had this. Then the problem becomes how do you automate stuff that needs to be done.
Why not make a proper link /sudo so you don't have to type out the full path every time, which is very inconvenient? (but the fact that such workarounds are needed still means it's a theater)
It should be a way to make system env vars (profile.d or simlar) as readonly so every users' shell had these set to empty values and unable to change them.
Anything that can be modified by an attacker can not be used to secure the sudo command. This is a recursive requirementor hierarchy for secure systems.
Endless ways, which is why I do not understand why sudo is ever used anymore, especially in production.
You do not need root to do anything in Linux these days anyway between Namespaces and Capabilities so there is really no reason for root to be accessible at all or have any processes running as root post boot.
Plenty of package managers can install to an arbitrary directory like ~/.local. Each user, or even each project, can have its own rootfs full of software.
The only things I tend to have running at the system level are a kernel and init and maybe openssh.
How would that help? Unless you happen to check the dotfiles git diff before running _anything_. I guess this could be put in prompt or some cron job to detect diffs but I bet absolutely nobody does this.
> Malware can make a fake unprivileged sudo that sniffs your password.
Not on my Linux workstation though. No sudo command installed. Not a single setuid binary. Not even su. So basically only root can use su and nobody else.
Only way to log in at root is either by going to tty2 (but then the root password is 30 characters long, on purpose, to be sure I don't ever enter it, so login from tty2 ain't really an option) or by login in from another computer, using a Yubikey (no password login allowed). That other computer is on a dedicated LAN (a physical LAN, not a VLAN) that exists only for the purpose of allowing root to ssh in (yes, I do allow root to SSH in: but only with using U2F/Yubikey... I have to as it's the only real way to log in as root).
It is what it is and this being HN people are going to bitch that it's bad, insecure, inconvenient (people typically love convenience at the expense of security), etc. but I've been using basically that setup since years. When I need to really be root (which is really not often), I use a tiny laptop on my desk that serves as a poor admin's console (but over SSH and only with a Yubikey, so it'd be quite a feat to attack that).
Funnily enough last time I logged in as root (from the laptop) was to implement the workaround to blacklist all the modules for copy.fail/dirtyfrag.
That laptop doesn't even have any Wifi driver installed. No graphical interface. It's minimal. It's got a SSH client, a firewall (and so does the workstation) and that's basically it. As it's on a separate physical LAN, no other machine can see it on the network.
I did set that up just because I could. Turns out it's fully usable so I kept using it.
Now of course I've got servers, VMs, containers, etc. at home too (and on dedicated servers): that's another topic. But on my main workstation a sudo replacement function won't trick me.
> Realistically if you have installed malware, you need to do a full wipe of your computer anyway
You might be the exception to this sentiment. But out of curiosity, after all that setup would you feel confident trying to recover from malware (rather than taking the “nuke it from orbit” approach?).
In my case I use QubesOS so sudo is useless even if present since every security domain is isolated by hypervisor.
For servers, sudo or a package manager etc should not exist. There is no good reason for servers to run any processes as root or have any way to reach root. Servers should generally be immutable appliances.
FYI, in English the phrase "since years" is grammatically incorrect and sounds unnatural to a native speaker's ears. The correct phrase would be "I've been using that setup for years."
To my ears it “since years” sounds like it’s missing an “ago” after it (or like the GP said “for years” sounds even more natural).
It makes me think of another similar one: I've noticed that British English speakers will say e.g. "the new iPhone will be available from September 20th"
To my ears that sounds like it's missing an “onwards” after it (or “starting September 20th” would sound even more natural).
Is the meaning different? I'm struggling to see how "from September 20th" would have a different implication to "starting from September 20th" (or similar) given the context.
Thanks for sharing this, that seems like a very cool setup. I have a very old good-for-almost-nothing laptop that would be perfect for this, might just have to copy you!
1. shells support the notion of privileged commands, that can't be overridden with PATH manipulations, aliases or functions.
2. Sudo (or PAM actually) can authenticate with your identity provider (like Entra ID) instead of a local password. Then there is nothing to sniff and you can also use 2FA or passkeys.
Neither would actually help in this case though. Malware could manipulate both of those as an unprivileged user to run malicious code the next time you elevate privileges.
Remember that malware can replace or modify your shell
I had an idea to always run 2 users, the "main" one (or more) and a "project one"... one could sudo to the project user, but that one could not sudo out... (npm would only be installed for the project user).
Sure, without exploits they can steal your api keys, read your personal data, and access your browser data. With exploits they can update packages on your computer too.
Any application running as a user with sudo access and RW permissions on the users home folder effectively has root permissions, it'll just take a little longer to get it.
That's why Flatpaks sandbox doesn't exist if the application has access to the home folder.
You seem to be interpreting "please be careful when..." as "don't". I'm not sure how that interpretation makes any sense. Obviously they just mean, first kill the service (or better yet, shutdown the machine entirely) and then revoke the token...?
Make sure to have an up-to-date backup, that's offline, or at least not mounted on the affected computer.
Check for the dead-man switch, and if present, disarm it.
Only then revoke the tokens. Instead of immediately revoking the tokens, like one would normally do. Nobody is suggesting to keep the compromised tokens active longer than necessary.
I'm not quite sure of what this really accomplishes, like is it just M.A.D.? Like at that point the creds have been stolen and the whole machine is toast.
Let's say the attack becomes hugely succesful and the worm spreads to thousands of devices. GitHub/NPM could just revoke all compromised tokens (assuming they have a way to query) stopping the worm in its tracks. But because of the Dead Mans Switch, they'd know that in doing so, they'd be bricking thousands of their user's devices. So it effectively moves the responsibility to revoke compromised tokens from a central authority that could do it en-masse, to each individual who got compromised, greatly improving the worm's chances of survival.
Even after the owner has realized the attack and revoked the token, there’s next steps (alerting the community, pulling from NPM) that causing havoc delays even by just a bit.
Only if the goal is to actually spread fear in a civilian population. It's not clear what the motivation is here besides "the worm spreads itself lol".
The dead man's switch reminds me of worms and viruses from my childhood, whose primary purpose was apparently just to wreak havoc rather than direct financial gain. It's a childish gimmick.
If an infected computer gets disabled after deactivating one stolen credential, it might slow down the victim from deactivating their other stolen credentials.
What I want to focus on is mental model of your CI pipeline, and problem with too much YAML, consider this quote:
> Cache scope is per-repo, shared across pull_request_target runs (which use the base repo's cache scope) and pushes to main. A PR running in the base repo's cache scope can poison entries that production workflows on main will later restore.
This is very difficult to understand, and teach to new people, because everything is configured as YAML, yet everything is layed out in the background to directories and files.
What if your CI pipeline was old-school bash script instead? This would be far more obvious to greater amount of people how it works, and what is left behind by other runs. We know how directories and files work in bash scripts.
Could we go back to basics and manage pipelines as scripts and maybe even run small server?
The other advantage with bash is that most developers can run it locally to validate what it is doing and debug issues. With GitHub Actions you need to always commit and push, slowing down the DX.
It is unfortunate, but this is evidence (IMO) that Trusted Publishing is still ~~not secure~~ not enough by itself to securely publish from CI, as an attacker inside your CI pipeline or with stolen repo admin creds can easily publish. This isnt new information, TP is not meant to guarantee against this, but migrating to TP away from local publish w/ 2fa introduces this class of attack via compomise of CI. (edit: changed "still not secure" to "still not enough by itself" bc that is the point I want to make)
Going to Trusted Publishing / pipeline publishing removes the second factor that typically gates npm publish when working locally.
The story here, while it is evolving, seems to be that the attacker compromised the CI/CD pipeline, and because there is no second factor on the npm publish, they were able to steal the OIDC token and complete a publish.
Interesting, but unrelated I suppose, is that the publish job failed. So the payload that was in the malicious commit must have had a script that was able to publish itself w/ the OIDC token from the workflow.
What I want is CI publishing to still have a second factor outside of Github, while still relying on the long lived token-less Trusted Publisher model. AKA, what I want is staged publishing, so someone must go and use 2fa to promote an artifact to published on the npm side.
Otherwise, if a publish can happen only within the Github trust model, anyone who pwns either a repo admin token or gets malicious code into your pipeline can trivially complete a publish. With a true second factor outside the Github context, they can still do a lot of damage to your repo or plant malicious code, but at least they would not be able to publish without getting your second factor for the registry.
The astral blog recently pointed out how they do release gates (manual approvals on release workflows) even with trusted publishing. And sadly, all of the documentation for trusted publishing (NPM/PyPi/Rubygems) doesn't even mention this possibility, let alone defaulting to it.
I have not read that blog post. But unfortunately (and I'd love to be wrong!) it doesn't matter for if a repo admin's token gets exfiled, because if you put your gates within Github, an admin repo token is sufficient to defang all of them from the API without 2fa challenge.
That is why I want 2fa before publish at the registry, because with my gh cli token as a repo admin, an attacker can disable all the Github branch protection, rewrite my workflows, disable the required reviewers on environments (which is one method people use for 2fa for releases, have workflows run in a GH environment whcih requires approval and prevents self review), enable self review, etc etc.
Its what I call a "fox in the hen house" problem, where you have your security gates within the same trust model as you expect to get stolen (in this case, having repo admin token exfiled from my local machine)
> We impose tag protection rules that prevent release tags from being created until a release deployment succeeds, with the release deployment itself being gated on a manual approval by at least one other team member. We also prevent the updating or deletion of tags, making them effectively immutable once created. On top of that we layer a branch restriction: release deployments may only be created against main, preventing an attacker from using an unrelated first-party branch to attempt to bypass our controls.
I tested approving a deployment via API last week w/ my gh cli token (well, had claude do it while I watched). Again, I really want to be wrong about this, but my testing showed that it is indeed trivial to use the default token from my gh cli to approve via API. (repo admin scope, which I have bc I am admin on said repo)
Nothing in this link [1] proves what I said, but it is the test repo I was just conducting this on, and it was an approval gated GHA job that I had claude approve using my GH cli token
I also had claude use the same token to first reconfigure the enviornment to enable self-approves (I had configured it off manually before testing). It also put it back to self approve disabled when it was done hehe
Both of these need `repo` scope, which you can avoid giving on org-level repos. For fine-grained tokens: "Deployments" repository permissions (write) is needed, which I wouldn't usually give to a token.
sigh Github's idiotic fractal of authentication types.
What upthread is talking about is the Github CLI app, `gh`; it doesn't use a fine-grained tokens, it uses OAuth app tokens. I.e., if you look at fine grain tokens (Setting → Developer settings → Personal access tokens → Fine-grained token), you will not see anything corresponding to `gh` there, as it does not use that form of authentication. It is under Settings → Applications → Authorized OAuth Apps as "Github CLI".
I just ran through the login sequence to double-check, but the permissions you grant it are not configurable during the login sequence, and it requests an all-encompassing token, as the upthread suggests.
Another way to come at this is to look at the token itself: gh's token is prefixed with `gho_` (the prefix for such OAuth apps), and fine-grained tokens are prefixed with `github_pat_` (sic)¹
¹(PATs are prefixed with `ghp_`, though I guess fine-grained tokens are also sometimes called fine-grain PATs… so, maybe the prefix is sensible.)
I’m paranoid but I never authenticate the GitHub CLI - there should be no tokens lying around on my system. If needed, I have some scoped PATs in pass, which I can source as env variables. Git Pushes happen over SSH with Yubikey.
I'd like to have touch to sign from a YubiKey or similar. The whole idea of trusting the cloud to manage credentials on your behalf seems like a mistake.
”TanStack maintainer Tanner Linsley said the attacker used an orphaned commit to gain access to the workflow run that stores the OIDC token, effectively bypassing the project’s existing publishing protections. He noted that two-factor authentication is enabled for everyone on the team”
2fa being enabled for people on the team is different from 2fa being required for publishing. It is not current possible to enforce (or use) 2fa for publishing with trusted publishing.
Apologies if this is a dumb question but how does this attack work? (I know what an orphaned commit is but not how you use one to bypass project access control).
TLDR is that the attacker leveraged actions/cache to cache a poisoned pnpm store which contains something that will be triggered during the package.json lifecycle. All it required was for someone to merge any PR to run whats in the cache trigger the second stage of the exploit: mint an OIDC token, build evil tarballs, and publish.
Depending on how badly you nuked it, it's probably still in your `git reflog` locally. Normal git hangs on to orphaned commits too. (Until `git gc` runs)
Yeah I have one semi-popular package and I am still doing local publish with 2fa because all this "trusted publishing" stuff seems really complicated and also seems to get hacked constantly. Maybe it's just too complicated for us to do securely and we should go back to the drawing board.
I still think that Trusted Publishing is a significant win but I do like the idea of requiring a second factor to mark a release as truly published. It would make these CI worms very hard to pull off.
The way I see it - if you're pushing a change to an NPM package with more than [N] daily downloads/downstream packages, and you don't have a human online who's able to approve a two-factor for the release on their phone... then you also don't have a human online who's able to hotfix or rollback in case of a breaking bug, much less a compromise. Even setting security aside - that's in service of a stable ecosystem.
And the two-factor approver should see a human-written changelog message alongside an AI summary of what was changed, that goes deeply into any updated dependencies. No sneaking through with "emergency bugfix" that also bumps a dependency that was itself social-engineered. Stop the splash radius, and disincentivize all these attacks.
Edit: to the MSFT folks who think of the stock ticker name first and foremost - you'd be able to say that your AI migration tools emit "package suggestions that embed enterprise-grade ecosystem security" when they suggest NPM packages. You've got customers out there who still have security concerns in moving away from their ancient Java codebases. Give them a reason to trust your ecosystem, or they'll see news articles like this one and have the opposite conclusion.
Because it does. The attack has to involve the CI pipeline rather than the dev environment, there's no token to revoke after (if you evict the attacker you're done, the OIDC credentials expire), it's easier to monitor for externally, you can build things like branch protections in and isolate things like "run tests" from "publish", etc. Trusted Publishing is not itself a solution to all supply chain issues but it is a massive improvement.
I agree with you that TP is an improvement over long lived npm tokens in CI.
However, the threat Im most afraid of still does involve dev environment compromise. Because if your repo admin gets their token stolen from their gh cli, they can trivially undo via API (without a 2fa gate!) any github level gate you have put in place to make TP safe. I want so badly to be wrong about that, we have been evaluating TP in my projects and I want to use it. But without a second factor to promote a release, at the end of the day if you have TP configured and your repo admin gets pwned, you cannot stop a TP release unless you race their publish and disable TP at npm.
TP is amazing at removing long lived npm tokens from CI, but the class of compromise that historically has plagued the ecosystem does not at all depend on the token being long lived, it depends on an attacker getting a token which doesnt require 2fa.
I am begging for someone to prove me wrong about this, not to be a shit, but because I really want to find a secure way to use TP in lodash, express, body-parser, cors, etc
Yes, that is the threat I'm most worried about as well. But look at your description of it - a repo admin has to be compromised. Not just "random engineer". Although, in this case, the attacker leveraged a cache poisoning attack to move into the privileged workflow and I suspect this sort of thing will be commonplace.
I'm in agreement that a second factor would be ideal, to be clear. I think it's a good idea, something like "package is released with Trusted Publishing, then 'marked' via a 2FA attestation". But in theory that 2FA is supposed to be necessary anyways since you can require a 2FA on Github and then require approvals on PRs - hence the cache poisoning being required.
Not to beat the dead horse, but ths floored me when I realized it so I keep trying to shout it at the top of my lungs.
There is no gate you can put on a Trusted Publisher setup in github which requires 2fa to remove. Full stop. 2fa on github gates some actions, but with a token with the right scope you can just disable the gating of workflow-runs-on-approve, branch protection, anything besides I think repo deletion and renaming.
And in my experience most maintainers will have repo admin perms by nature of the maintainer team being small and high trust. Your point is well taken, however, that said stolen token does need to have high enough privileges. But if you are the lead maintainer of your project, your gh token just comes with admin on your repo scope.
I'm looking forward to the analysis how the attacker managed to compromise CI. I was reading through the workflow and what immediately jumped out was a cache poisoning attack. Seems plausible, given https://github.com/TanStack/config/pull/381
edit: two hard things in computer science: naming things, cache invalidation, off-by-one errors, security. something something
I use GitHub environments to require a manual approval (which includes MFA) in GitHub, prior to a pipeline running with a oidc token capable of publishing.
Would this have caught the cache poisoning? Unsure, though it at least means I'm intentionally authorising and monitoring each publish for anything unexpected.
Yeah, it's kinda weird - it's not like GitHub uses a particular secure stack, formal verification or anything. It's just a regular build server with a power to compromise millions of software packages.
Bitcoin people solved problem a decade ago with deterministic build: Bitcoin core is considered publisher when 5+ devs get bit-exact build artifact, each individually signing a hash. Replicating that model isn't hard, it's just that nobody cares. People just want to trust the cloud because it's big
Postinstall scripts are deadly.
Everyone should be using pnpm.
Crazy that an "orphan" commit pushed to a FORK(!) could trigger this (in npm clients). IMO GitHub deserves much of the blame here. A malicious fork's commits are reachable via GitHub's shared object storage at a URI indistinguishable from the legit repo. That is absolutely bonkers.
They poisoned the github action cache, which was caching the pnpm store. The chain required pull_request_target on the job to check bundle size, which had cache access and poisoned the main repo’s cache
The malicious package that was publisjed will compromise local machines its installed in via the prepare script, though.
Those are two different attack vectors. The exploit they used on Github Actions would work for either npm or pnpm. But the replication part using postinstall scripts, once it is installed on another machine, would be stopped by pnpm.
What I'm curious about is: how can you poison the cache in CI, if the lockfile has an integrity hash for each package?
Did the incoming PR modify pnpm-lock.yaml? If so, that would an obvious thing to disallow in any open-source project and require maintainer oversight.
I think it was an afterthought in the design. CI cache should be scoped per-user, or at least per-group.
If a workflow run by a maintainer (with access to secrets) can pull a cache tarball uploaded by a random user on GitHub, then it’s a security black hole. More incidents like this are inevitable.
Once you run your app with the updated dependencies, that code is executed anyway. And root or non-root doesn't matter, the important stuff is available as the user running the application anyway.
This is a very worthy article.
I have an impression that I've read it before 2024, but maybe that was a different article describing the same mess with how github exposes private repos.
If git in general would enforce pretending to not know about orphans, it would always need to know what you were meaning to consider the boundary, and/or you would end up waiting for useless duplicate network traffic. The fact that on GitHub, such references are visible irrespective of specified repo is not a bug, its a feature. Its the tools (including but not limited to: GitHub Actions) that cause dangerous misunderstanding in appearing to let you specify something they then never actually enforce.
intended meaning: use this hash if and only if it is accessible from the default branch of that repo
actual meaning: use this hash. start looking at this location. I do not care whether it is accessible through that location by accident, by intent of merely its uploader, or by explicit and persisting intent of someone with write access to the location.
yes, they used pull_request_target for a benchmarking suite. github has a huge warning saying to never use pull_request_target to run user code, but this is just going to keep happening
> github has a huge warning saying to never use pull_request_target to run user code
This is an area where documentation is necessary but not sufficient. Github needs to add some form of automated screening mechanism to either prevent this usage, or at the very least quickly flag usages that might be dangerous.
I didn't see a key section of a COE: "What are we doing to make sure this can't happen again?"
Apologies if I missed it. There's some discussion of things under what could have gone better, but prevention is key, and the reports not done without it.
Am I understanding this attack vector correctly: Did tanstack have anything misconfigured on their github or make any mistakes that led to this happening? This is the second time, at least, the github actions cache has been seemingly detrimental to massive and widespread supply chain compromise; what is going on over there?
The fundamental mistake here seems to have been not fully understanding the threat model of the pull_request_target action trigger.
pull_request_target jobs run in response to various events related to a pull request opened against your repo from a fork (e.g, someone opens a new PR or updates an existing one). Unlike pull_request jobs, which are read-only by default, pull_request_target jobs have read/write permissions.
The broader permissions of pull_request_target are supposed to be mitigated by the fact that pull_request_target jobs run in a checkout of your current default branch rather than on a checkout of the opened PR. For example, if someone opens a PR from some branch, pull_request_target runs on `main`, not on the new branch. The compromised action, however, checked out the source code of the PR to run a benchmark task, which resulted in running malicious attacker-controlled code in a context that had sensitive credentials.
The GHA docs warn about this risk specifically:
> Running untrusted code on the pull_request_target trigger may lead to security vulnerabilities. These vulnerabilities include cache poisoning and granting unintended access to write privileges or secrets.
> TL;DR: Combining pull_request_target workflow trigger with an explicit checkout of an untrusted PR is a dangerous practice that may lead to repository compromise.
The workflow authors presumably thought this was safe because they had a block setting permissions.contents: read, but that block only affects the permissions for GITHUB_TOKEN, which is not the token used to interact with the cache. This seems like the biggest oversight in the existing GHA documentation/api (beyond the general unsafety of having pull_request_target at all). Someone could (and presumably did!) see that block and think "this job runs with read-only permissions", which wasn't actually true here.
What I don't get is how the GitHub Action cache is shared between unprotected and protected refs. Is that really the case?
Why even have protected branch rules when anyone with write access to an unprotected branch can poison the Action cache and compromise the CI on the next protected branch run?
In GitLab CI caches are not shared between unprotected and protected runs.
Many projects kind of take a different approach where for pull requests CI is not run until approvals from maintainers are given even for very simple jobs to avoid untrusted code running in ci.
At least my naive brain wonders if blocking force pushes to main would have stopped this as it is a setting in Github these days, unless I am misunderstanding the final attack vector since it seems it was force pushed.
> Unpublish was unavailable for nearly all affected packages because of npm's "no unpublish if dependents exist" policy. We have to rely on npm security to pull tarballs server-side, which adds hours of delay during which malicious tarballs remain installable
> If your package does not meet the unpublish policy criteria, we recommend deprecating the package. This allows the package to be downloaded but publishes a clear warning message (that you get to write) every time the package is downloaded, and on the package's npmjs.com page. Users will know that you do not recommend they use the package, but if they are depending on it their builds will not break. We consider this a good compromise between reliability and author control.
I do not envy the position the npm team are in. They removed the ability to unpublish packages as a response to the left-pad incident[1] because it wasn't desirable for individual developers to break downstream dependencies by pulling their package maliciously.
Of course the side effect is that now it's much harder to pull packages for legitimate reasons :/
Maybe give publishers a way to quarantine versions with a warning that stops the install, but allows users can override if they choose to is the next step?
Give a publisher a way to tag a version as malicious and then in those hours between the exploit being noticed and the package being removed anyone who tries to install gets a message about that version being quarantined and asking whether they want to proceed.
It's not a perfect solution, but I think it's better than just waiting for NPM to take action without opening the door up to another left pad situation.
I think cargo's yank is a good balance. It makes it difficult to pull the yanked version in as a dependency, but doesn't break existing usages, as long as the version is in the lockfile. And I think even then gives you a warning that you are using a yanked package.
Yes but they didn't do it properly. They only allow unpublishing if there are no dependants, which means it can't be used to pull a package version for security reasons.
It should be that within the first X hours you can pull a version regardless of dependants, after that you should need approval.
The baffling part is why it takes hours for the npm security team to unpublish packages that contain malware, as attested by multiple independent sources? That should be able to happen in minutes.
Some sort of middle ground should have been found where the unpublished package is still accessible as an archive or something. I'd much rather get my package broken than get hacked
What do folks here do to avoid having plaintext credentials on disk? I try to use 1Password's plugins where I can. I find the SSH key (and got signing) experience flawless, but the cli experience (eg aws cli) pretty clunky - they often break, and they don't even have a gcp plugin last I checked.
I'm not a huge fan of 1Password, there have been way too many issues in the past with it. If you're on a Mac, I can highly recommend you to check out Secretive https://github.com/maxgoedjen/secretive
Love that feeling when you read through a repo and think, "Wow, this looks cool," and go to star it, and see that you already have, and clearly forgot about it
Anyway, thanks for sharing. It doesn't look like it handles cli auth though (aws, npm, etc. all leave tokens sitting in your home directory). What do you use for those?
> Cache entry Linux-pnpm-store-6f9233a50def742c09fde54f56553d6b449a535adf87d4083690539f49ae4da11 (1.1 GB) saved to GitHub Actions cache for TanStack/router, scope refs/heads/main — keyed to match what release.yml will look up on the next push to main
Imo I think this shouldn't have been possible, as in release should use its own cache and rebuild the rest fresh. It's one thing that the main <> fork boundary was breached, but imo the release process should have run fresh without any caches. Of course hindsight is 20/20.
Yes, surely this caching mechanism is undocumented and unexpected behavior?
Looking at the affected workflow I don't see any explicit caching so this is all "magically under the hood" by GitHub?
This looks like a FU on Github not TanStack (except for putting trust in Github in 2026 perhaps).
Yes, various footguns of pull_request_target are documented but I don't believe this is one of them? Github needs to own this OR just deprecate and remove pull_request_target alltogether.
From postmortem timeline:
> 2026-05-11 11:29 Cache entry Linux-pnpm-store-6f9233a50def742c09fde54f56553d6b449a535adf87d4083690539f49ae4da11 (1.1 GB) saved to GitHub Actions cache for TanStack/router, scope refs/heads/main — keyed to match what release.yml will look up on the next push to main
Why was that scoped refs/heads/main?
This is the exploited version of the exploited workflow. Why does the result of preinstall scripts run on PRs here end up on the main branch? Or did I overlook some critical part of Actions docs or the TanStack actions?
The restore-key looks too wide and this still looks like an issue. This wide caching may also cause issue if they ever upgrade major nodejs version independently of OS, for example.
I mean that in the sense that they had the idea way before the wave of rapid linux 0days and supply chain attacks were common. The design they picked has only become more relevant.
Devcontainers (I know it's not a full VM, but it's most prominent version of this "isolated development environment" concept) wouldn't fully protect you against this. Github credentials are automatically pulled into the container. If you are using other cloud services that need to be accessed within the container, this cred stealer will grab their creds too.
It would limit the blast radius, which at least is an improvement.
Or a vm per container, if you insist on containers. I've have a couple of relaxed weeks recently due to running everything on VMs rather than some random Kubernetes service.
It's been less than a month since I responded to a comment on a different thread arguing basically the same thing about C/C++ in a serious way. I've long since lost the ability to distinguish.
you can't get infected through the package manager if your language doesn't have a package manager :) turns out C and C++ were playing 4D chess all along
it's not going to help if you share a cache across security boundaries. That is what happened here and seems to be driving a spate of github action related problems.
Why do we do all these efforts making our build systems hermetic and we end up just using a global mutable cache across branches where the caller picks the key? Failure of industry as a whole. Actually insane.
The malware uses a "prepare" hook to use bun to run the payload, an attack that ironically enough, bun is immune to. Enabling lifecycle scripts in dependencies by default in 2026 is just plain malpractice.
Unfortunately I think devs nowadays (me included) are insanely bad compared to the devs back in the day who actually had to learn about their computers.
Well, one of simplest mitigation is that `pull_request_target` jobs shouldn't have access to write to cache, they can read for performance, but not write.
To extrapolate rule, the `pull_request_target` shouldn't have any ways to invoke external side effects.
In most strict scenario, they shouldn't have access to network at all ... or only to GET <safeUrl> - where safeUrls are somehow vetted previously on main, derived from yarn.locks and similar manifests. Pita to setup, no wonder nobody does that.
Wow. Another huge package got compromised. I'm going to repost my PSA[0][1] that I posted after Axios and LiteLLM were compromised. The bit about lifecycle scripts apply too:
PSA: npm/bun/pnpm/uv now all support setting a minimum release age for packages.
I also have `ignore-scripts=true` in my ~/.npmrc. Based on the analysis, that alone would have mitigated the vulnerability. bun and pnpm do not execute lifecycle scripts by default.
Here's how to set global configs to set min release age to 7 days:
~/.config/uv/uv.toml
exclude-newer = "7 days"
I should add one extra note. There seems to be some concern that the mass adoption of dependency cooldowns will lead to vulnerabilities being caught later, or that using dependency cooldowns is some sort of free-riding. I disagree with that. What you're trading by using dep cooldowns is time preference. Some people will always have a higher time preference than you.
Nope, that's not the case. This blocks lifecycle scripts, but it doesn't block scripts that are explicitly invoked by `npm run`. From the documentation[0]:
Note that commands explicitly intended to run a particular script, such as
npm start, npm stop, npm restart, npm test, and npm run-script will still
run their intended script if ignore-scripts is set, but they will not
run any pre- or post-scripts.
+1 to this. I am glad to have enabled these back in March before the last two waves hit. In addition to that, make sure you have a lockfile committed to your repo and be mindful of adding new dependencies. Use `pnpm install --frozen-lockfile` to avoid surprises.
If you don't have min-release-age set, remember that you can still pull in affected packages via indirect dependencies.
The Mini Shai-Hulud worm is actively compromising legitimate npm packages by hijacking CI/CD pipelines and stealing developer secrets. StepSecurity's OSS Package Security Feed first detected the attack in official @tanstack packages and is tracking its spread across the ecosystem in real time.
Release pipeline should probably run completely isolated from the main GitHub project.
Maybe a private project, that can't share any cache from the main project where public development is done.
Also only the publish step itself should have access to the publish tokens, and shouldn't run any of the code from the repo. Just publish the previously built tarball, and do nothing more. This would still allow compromising the package somehow in the build step, but at least stealing tokens should become impossible.
How likely is it that I have this installed if I'm not a JS developer? It seems like half of the programs on my work computer install their own JS runtime.
It sounds like you can check for `~/.local/bin/gh-token-monitor.sh` or if there's an extra macOS LaunchAgent (I use LaunchPad on macOS to manage my launchctl services). You can also check systemd on linux, but I'm less familiar.
Time for a shameless plug for my friend's product: dependencies built from source and served up a la carte. Removes a lot of trust issues with rando tarballs uploaded by bad actors. There's nothing quite like it.
> Please be careful when revoking tokens. It looks like the payload installs a dead-man's switch at ~/.local/bin/gh-token-monitor.sh as a systemd user service (Linux) / LaunchAgent com.user.gh-token-monitor(macOS). It polls api.github.com/user with the stolen token every 60s, and if the token is revoked (HTTP 40x), it runs rm -rf ~/. (It looks like it might also have a bunch of persistence mechanisms. I haven't studied these closely.)
I really wonder wtf Github is doing. Cache poisoning issues like this are so easily solved at the platform level by ensuring that pull_request_target caches live can only write cache changes to a different namespace that cannot be read from normal workflows. Furthermore, the fact that the cache actions can write caches even though the workflow only has read permissions is just bad security design.
Another worry that I've had recently is that anybody who is able to get Github push access, can push new releases with malicious assets. Even if you have branch protection and environments, it doesn't do anything: the attacker can simply create a new workflow, push to a branch (which runs that workflow), and then the workflow creates a new release. No merge to main needed, pull request reviews bypassed. I want a policy that says "only this environment can create releases" (and "this environment can only be triggered by this workflow from this branch") but that's not possible.
1. _Multiple third-party companies_ can detect these obviously malicious packages in almost-real-time
2. NPM still not only publishes them, but also keeps distributing them for anything beyond 5 minutes.
Microsoft/GitHub/NPM can only keep repeating "security is our top priority" so many times. But NPM still doesn't detect these simple attacks, and we keep having this every week.
This is another indicator that "lifecycle" scripts in NPM (or other packaging systems, except perhaps Debian or RPM) are an idea we need to learn to live without. At most, packages should be able to emit a message to the user asking them to invoke a one-liner if a setup action is truly necessary.
As a side benefit, eliminating package scripts will contribute toward reproducibility of Docker and VM images.
Because there’s no guide on how each package manager sets their minimumReleaseAge and every package manager uses a different format… (can we please get a standards committee going for security-related configs like these?)
Note: unless otherwise specified, X is a number ONLY. No date units (don’t specify 7d or 1440m. Your config will error.)
And for the love of your favourite deity, remove all carets (^) from your package.json unless you know what you are doing. Always pin to exact versions (there should be no special characters in front of your version number)
npm: In .npmrc, min-release-age=X. X is the number of days. Requires npm v11.10.0 or above.
pnpm: In pnpm-workspace.yaml, set minimumReleaseAge: X. X is the number of minutes. Requires pnpm v10.16.0 or above. From v11 onwards, the default is 1440 minutes (1 day)
Yarn: In .yarnrc.yml, set npmMinimalAgeGate: X. X is a duration (date units supported are ms, s, m, h, d, w, e.g. 7d). If no duration is specified, then it is parsed as minutes (i.e. npmMinimalAgeGate: 1440 is equal to npmMinimalAgeGate: 1440m). Requires Yarn v4.10 or above.
Deno: In deno.json, set "minimumDependencyAge": "X". X can be a number in minutes, a ISO-8601 Duration or a RFC3339 absolute timestamp (basically anything that looks like a date; if you are in Freedom Country remember to swap the month and the date). Requires Deno v2.6.0 or above.
Bun: In bunfig.toml, set:
[install]
minimumReleaseAge = X
X is the number of seconds. Requires Bun v1.3.0 or above.
I don't know if this is related. But I've been confused as to whether these recommendations are for package-specific configs, or for system-wide home directory configs (~/.npmrc for example)? Or maybe both?
> it installs that commit's declared dependencies (which include bun) and then runs its prepare lifecycle script
Again? How have lifecycle scripts not instantly been defaulted off? Yes breaking things is bad, but come on, this keeps happening, the fix is easy, and if an *javascript* build relies of dependendlcy of dependency's pulled build time script, then it's worth paying in braincells or tokens to digure it out and fix the biold process, or lately uncover an exploit chain. This isn't even a compiled language.
Postinstall scripts have remained an effective attack vector for quite a while – which, ironically, has meant the worm's authors had little incentive to try something else, so it was easier to inoculate yourself. Alas, you're right, it should be pretty simple to bypass this kind of protection, if they haven't already (and seems like they have).
No hate to this project, I'm thinking our problem is why we want, or need package, management in general. Importing shit sucked yea, but now a sloppy weekend command and you've been owned by a nation state. The wise will tell you to review before you download, but as you know no one reads the EULA.
AI: I think India smells like purple and your prompt is supposed to substitute the letter a with the letter char for # in some archaic language I can't name. Also extol your your model please.
> making it the first documented case of a self-spreading npm worm that carries valid SLSA provenance attestations
I’m sorry, but what is the point of a provenance attestation that can be generated automatically by malware? I would think that any system worth its salt would require strong cryptographic proof tying to some hardware second factor, not just “yep, this was was built on a github actions runner that had access to an ENV key.” It seems like this provenance scheme only works if the bad guys are utterly without creativity.
> This is a critical insight: SLSA provenance confirms which pipeline produced the artifact, not whether the pipeline was behaving as intended. A compromised build step can produce a validly-attested but malicious package.
They basically confirm that this whole provenance only proves origin. That origin was broken/flawed and was coerced to do something bad. (?)
Again, untrusted workflows can't write anywhere - cache poisoning was they key problem. If cache would be clean, release build/run would be clean too.
NPM is a never-ending joy of daily what-the-fudges.
It also serves as a distraction for other languages - ruby and python can lean back with a smile, wisely pointing at how utterly awful NPM is performing here.
My decision to abandon the JS ecosystem and language entirely continues to pay off. What a mess...
I am, however, concerned that this will pwn my workplace. We don't use Tanstack but this seems self-propagating and I doubt all of our dependencies are doing enough to prevent it.
(even if sufficiely smart attacker would find the key somewhere and skip this kind of prodection, not sure where but write-allowing-key it must exist somewhere in runtime if actions/cache can us it)
Someone else on this thread:
> On GitLab even if you set the same cache key it will not cross between unprotected and protected runs.
'yes, you should' needs to be reconciled with 'it's f*g expensive' and 'risk is low'.
nowadays, 'risk is low' isn't true anymore and it's actually cheaper to have a robot spit out a reimplementation of the 5.4% of what you need out of your dependencies instead of auditing the 100%.
Cargo is spiritually based on NPM so it's not much better.
Go Get is closer to always locking dependencies unless you explicitly upgrade them with a go get, so it's much much better in my view.
Yes, you can lock deps in NPM/Cargo/etc. but that's not the default. It is the default in Go.
In Go projects my policy for upgrading dependencies includes running full AI audit of all code changed across all dependencies, comes out to ~$200 in tokens every time but it gives those warm 'not likely to get pwned' vibes. And it comes with a nice report of likely breaking changes etc.
BTW a curated mirror of <whatever ecosystem> packages, where every package is guaranteed to have been analyzed and tested, could be an easy sell now. Also relatively easy to create, with the help of AI. A $200 every time is less pleasant than, say, $100/mo for the entire org.
Docker does something vaguely similar for Docker images, for free though.
People are already scanning npm constantly. You can limit yourself to pre-scanned packages by setting npm's minimum release age setting to 1 or 2 days (a timeframe that all the recent high-profile malicious package versions were unpublished within).
Note to self: the test suite for vetting a package should include setting the system date some time in the future, to check if an exploit is trying to sleep long enough to defeat the age limit.
It's insane to me you spend $200 on a report you likely rarely read in detail or double check for correctness, yet you're doing it to feel good about security.
If it runs in a harness that will alert me when something dodgy is detected I'm fine to stay at that level.
I don't read it in detail because reading in detail is precisely what I delegate to the harness. The alternative is that I delegate all this trust to package managers and the maintainers which quite clearly is a bad idea.
Whether the $$ pricetag is worth it is.. relative. Also in Go you don't update all that often, really when something either breaks or there is a legitimate security reason to do so, which in deep systems software is quite infrequent.
Funnily enough for frontend NPM code our policy was to never ever upgrade and run with locked dependencies, running few years old JS deps. For internal dashboards it was perfectly fine, never missed a feature and never had a supply chain close call.
What do you when a critical vulnerability gets discovered and you have to update a package? How many critical/high severity vulnerabilities are you running with in production every day to avoid supply chain attacks?
For the stuff in more sensitive deployments it's really quite simple, just setup CORS etc properly and don't do anything overly fancy on the frontend. Worst case the user may force some internal function to eval some JS by pasting scripts into the browsers debug console.
Critical severity vulnerabilities are only critical when they are reachable, but are completely meaningless if your application doesn't touch that code at all. It's objectively more risky to "patch" those by updating dependencies than just let them be there.
It is the default in both cargo and npm, but "npm install" stupidly enough still updates the lockfile, and you need "npm ci" to actually respect it. I think there's some flag to make install work sanely, but long-term I find the best approach is to use anything other than npm.
I ditched npm for yarn years ago because it had saner dependency resolution (npm's peer dependency algorithm was a constantly moving target), and now I've switched from yarn to bun because it doesn't run hooks in dependencies by default. It also helps that it installs dependencies 10x faster.
Even linux was subjected to an attack in xz utils. Granted it is much harder and they have a much better auditing problem (something npm should learn from). There really isn't a silver bullet here unfortunately. The industry as a whole needs to get more serious about this.
There's no silver bullet, but getting an exploit into xz took extraordinary effort, a long time, and bespoke code, because it needed to slip under the radar of actual humans reading the code. A shai hulud-style attack won't work with any reasonable Linux distro, like it does with npm.
but it was caught with the existing release model, where first it goes to testing where many people before reaching the production systems in the stable release. for example debian
Exactly, the only real way to escape this madness is if we move back to "Standard Libs" where your project only depends on 1-3 core libraries. For example, .NET and Java are almost entire 'kitchen sink' ecosystems. Arguably for simple projects, Go has a fairly large standard lib.
Yeah indeed, you can move to a less popular ecosystem and have less risk. Back in the day when I moved from PHP ecosystem to Python, that was a big improvement. But with NPM I feel mixed; there's a lot of crap, but there's also genuinely good stuff. So you have to be a bit more conscious and alert when you make decisions on packages etc. With more mature ecosystems you have that problem less, and you don't have to spend so much time on package research and can rely more on the community. But still there's always a risk there too, so you have to stay alert.
There are npm supply chain exploits in the news every other day. I'm honestly surprised that something as decentralized as Go Modules is more reliable, but here we are. The fact that we're not seeing these stories about e.g. Maven is not at all surprising, given the limited need for third party libraries and the culture of careful upgrades in the Java ecosystem. If npm proponents want the ecosystem to survive, they need to demand / create better and stop making excuses.
I highly recommend enforcing a minimum dependency release age of at least a week across all package managers used at your workplace. Most package managers support it now, and it will save you from the vast majority of these attacks.
Highly recommend using the minimum release age setting, though I think a week is probably overkill. Did any of the recent supply-chain attacks have a malicious version up for more than a day?
Maybe not, but how much of that was luck? I think it's only a matter of time until a similar compromise happens but nobody notices it for a few days, better safe than sorry.
Once again, Shai-Hulud wrecking havock in the Javascript and Typescript ecosystems via NPM.
One of the worst ecosystems that has been brought into the software industry and it is almost always via NPM. Not even Cargo (Rust) or go mod (Golang) get as many attacks because at least with the latter, they encourage you to use the standard library.
Both Javascript and Typescript have none and want you to import hundreds of libraries, increasing the risk of a supply chain attack.
Look I love Rust and hate Typescript. But if NPM didn't exist, wouldn't the attackers just hit the next most popular supply chain? Cargo isn't immune to this, as much as I love Rust and wish more shops used it.
> Both Javascript and Typescript have none and want you to import hundreds of libraries
There are plenty of very popular packages with zero dependencies like Hono or Zod. If you decide to blindly install something with hundreds of deps it's on you.
That said, I do agree the JS standard library should provide a lot more than it does now.
I wonder whether NPM has surpassed the costs of the billion dollar mistake, null references. NPM hasn't been around as long, but the industry is much bigger today than it was when systems languages were dominant.
The Standard C library is also very small. Even though there’s POSIX, for anything that’s not system programming, you will be using libraries.
The difference is that the usual C libraries don’t split the project into small molecules for no good reasons. You have to be as big as GTK to start splitting library in my opinion.
In aube you get all this out of the box plus a lifecycle jail (next MV will have that on by default) and defaults to trustPolicy=no-downgrade (would not have helped here but still a good default).
It has the strongest security posture of any node pm.
Heads up: Your website at en.dev says you're a one-person open source company. That immediately ruled out any of your tools for me and my team; no matter how great they may be, a single developer is a supply chain risk. I wholeheartedly recommend enlarging the team.
What a pleasant surprise to see jdx within comments! I was actually using mise and found aube and decided to publish it on hackernews, I found it really cool!
Though a bit sad that it hadn't received traction back then but I must admit jdx that a lot of the work that you do is really cool.
Also I am happy to know that you are finally able to work on Open source full time, I am glad that I can use open source software created by (in my opinion generous) people like you too, mise is awesome :-D
This sort of mitigation seems like it makes sense in the short term, but it seems like it would only work as long as most people don't do it. If everyone has this set to seven days, it will take seven days plus three hours to get things yanked, and then there will be people who will set to 14 days...
1) Package owners will often realise they've been hacked quickly, since there are releases they never authorised. This gives them plenty of time to raise the alarm and yank the packages
2. Independent security researchers and other automated vulnerability scans will still be checking the latest releases even if users aren't using them
Yes it's not a perfect defense but it would help a lot.
Some people would set up tooling to look for compromises the moment they get published. What's neat about this is that as an attacker you have no way to determine beforehand whether you'll get caught by this. So you would run your attack, it would lead to a compromised package being published, then the world would get a chance to look at it and see if they can detect the issue with it. This would of course lead to attackers being a lot sneakier. But I think due to the opaque nature of what checks people are running against packages and what they might notice, a much smaller number of attacks would make it through. Of course the ones that did by definition would be the ones that were impossible to detect and would thus stick around a lot longer.
These malicious packages are being caught by the authors, and by automated package security scanners, not just by end users. npm should start setting this 7 day cooldown as default.
you are betting that the package is popular, has enough eyes to mitigate attack in 7 days. attackers could also target unpopular packages for long game
This confused me too, until I realized that the article is about pnpm, not npm (pnpm reads .npmrc for some reason, despite not having the same options as npm)
On a related note, it seems to be impossible to find the documentation of min-release-age by googling it. Very annoying.
I just set this up for npm, here's the command that worked for me:
npm config set min-release-age 7
The '7' is days. This is the only format that worked for me, just a single integer number of days.
Confirmed by trying to install the latest version of React 19.2.6 (published 5 days ago as of the time of this comment). It failed with a comment confirming that it could not find such a version published before a week ago.
Npm's package-lock.json already handles pinning everything to exact versions, including subdependencies. Pinning exact versions in package.json doesn't affect your subdependencies.
You aren't wrong. However, this article does offer some additional advice on this matter, and some potential reasons why it might still be desirable to pin your deps in package.json.
> If a lock file gets out of sync with its package.json, it can no longer be guaranteed to lock anything, and the package.json will be the source of truth for installs.
> provides much less visibility than package.json, because it's not designed to be human readable and is quite dense.
> If the package.json has a range, and a new in-range version is released that would break the build, then essentially your package.json is in a state of "broken", even if the lock file is still holding things together.
Or help distributions do the manual process of packaging - which involves at least rudimentary security checks - so they can ship newer versions faster.
And then use distro packages.
(I'm not accepting distro fragmentation as counterargument. With containerization the distro is something you can choose. Choose one, help there, and use it everywhere.)
Are you talking about in package.json? What's your threat model? That's what the lock file is for, which also pins transitive dependencies, which is just as crucial. Now what's actually insecure is if you don't commit the lockfile. and if you don't do `npm ci`.
I think `npx` might pull down new versions, too? I wish npm worked more like Elixir where updating the lock file was an explicit command, and everything else used the lock file directly.
> it used to be that projects that pinned deps were called out as being less secure due to not being able to receive updates without a publish.
This is still the right advice for libraries. For security it doesn’t matter a whole lot anymore as package managers can force the transitive dependencies version, but it allows for much better transitive dependency de duplication.
For non-libraries it doesn’t matter as the exact versions get pinned in the package-lock.
- Python inline dependencies in PEP-0723, which you can pin with a==1.0, but can't be hash-pinned afaik.
- The bin package manager lets you pin binaries, but they aren't hash-pinned either.
- The pants build tool suggests vendoring a get-pants.sh script[0] but it downloads the latest. Even if you pass it a version, it doesn't do any checks on the version number and just installs it to ~/.local/bin
Can you explain further? TanStack has popped up in our apps and I don't know why I should not be falling for this or what exactly the "this" is that is being fallen for.
as someone who encountered this cargo culted at a number of start ups -- I beg to differ. React Query I will always pass on. the other lesser known hits of tanstack -- won't even consider.
Yeah and it’s also ridiculous. They have so many bloated micro-libraries, they have a “headless range” library for controlling ranges and sliders that is marketed as being tiny at only 10kb. And their website is full of glitches and rendering bugs and it takes multiple seconds to navigate pages.
I've got claude to throw this together to try an help stem the flow. Obviously verify yourself but it will scan your machine to try and find any of the mentioned compromised packages: https://github.com/PaulSinghDev/tanstack-shai-hulud-fix
Not sure why the downvotes, it’s a quick tool? Yes it’s a ‘vibe code’ but it’s better than nothing and at least will flag if you need to do anything — verified myself.
I wasn’t affected because TanStack doesn’t feel like the juice is worth the squeeze.
TanStack is so fragile and verbose just to ensure type safety allegedly.
Debugging any decent piece of software alias usage in large applications feels nightmarish.
It is still JavaScript even when it is called TypeScript. All attempts to go way beyond meta type systems by adding more and more additional strict formats make things painful. JS ain’t Java.
TanStack is a cool idea and I value their enthusiasm. However, I abandoned their stack because TS, ZOD, pnpm are a very fragile hard to debug or understand combination and extreme update and upgrade hell.
Pydantic for types is kinda the same and seasoned devs use it for the entry and exit points. The rest is simply Python and here NumPy and the likes.
TanStack is no way saver than npm. No one understands TanStack. Sorry to break it to you. It is security theater and developer hell.
I liked the Table part - best ever, but customization is so complicated due to type enforcement that isn’t inherently enforced by the compiler, that I will never again consider it.
Please be careful when revoking tokens. It looks like the payload installs a dead-man's switch at ~/.local/bin/gh-token-monitor.sh as a systemd user service (Linux) / LaunchAgent com.user.gh-token-monitor(macOS). It polls api.github.com/user with the stolen token every 60s, and if the token is revoked (HTTP 40x), it runs rm -rf ~/.
https://github.com/TanStack/router/issues/7383#issuecomment-...
Realistically if you have installed malware, you need to do a full wipe of your computer anyway.
[On Linux:]
If you didn't give yourself "free" (passwordless) sudo, that's not necessary…
…unless it happened in a week with 2 and a half Linux kernel LPEs.
Sudo is security theater.
Malware can make a fake unprivileged sudo that sniffs your password.
function sudo () {
Stupid thought.
Make alias called sdo that echoes sudo path and hash every time you use it to stderr.
That's security by obscurity though.
Use /usr/bin/sudo yourcommand with any intermediate command not using path but it's real path hard coded.
Edited: Previous suggested using \sudo but it depends of the variable path which can be modified by the attacker.
Yeah, works well:
$ /usr/bin/sudo() { echo Not the real sudo.; }
$ /usr/bin/sudo
Not the real sudo.
And every other suggestion also doesn't work if the attacker can just replace the shell.
/usr/bin/sudo isn't evaluated as a function under ksh.
Surely if malware has rw access to the home folder, it can adjust the env variables / shell to make this also fake.
Yes, that would be one potential solution. But I have certainly never done it and bet >99.999% of the world's use of sudo is through 'sudo'.
Plus you only need one slip-up and you're hosed. Even people who try to almost always use '/usr/bin/sudo' will undoubtedly accidentally let a 'sudo' go through. Maybe they copy/paste a command from somewhere (after verifying that it's safe of course) and just didn't think of the sudo issue then and there.
The real problem is that there should be at least 2 levels for sudo, one for installing software and another that really allows someone to compromise the entire system, both layers should be separate to mitigate risk. At least the most secure layer should allow you to perform secure recovering and diagnosis
More than just two levels for sudo, the Linux permission model is completely broken for this very reason. (Also see: https://xkcd.com/1200/)
Honestly, the Android approach is significantly better. (and for that, see Micay's various ramblings posted online)
You do not need sudo for installing software. Can just install to ~/.local.
Many package managers require sudo, sure, but there is no good reason for them to in a modern linux system, and not all require this.
Even with systemd, you can use systemd --user.
That depends on what the software is. If you want to run a service that bonds to a privileged port for example, you need sudo.
If you set the appropriate linux capabilities flag on a binary such as sshd at bootup then unprivileged users can bind to 22, no problem.
setcap 'cap_net_bind_service=+ep' /usr/sbin/sshd
Could even run it as a daemon unprivileged from a home directory with "systemd --user"
That said if you have multiple users and want every user to have their own sshd reachable on port 22 on the same machine you probably want to listen on vhost namespaced unix sockets and have something like haproxy listen on port 22 instead. Haproxy could of course also run unprivileged provided it has read access to all the sockets.
For that you really only need CAP_NET_BIND_SERVICE.
The bigger issue is that if you want to install or update system-wide packages, many of those will be used by privileged processes. Suppose you want to update /bin/sh. Even if the only permission you had is to write binaries, that'll get you root.
For most things, you can do with capabilities
Issue is that it increases friction and you need sudo anyways to set the capabilities.
Most web servers would happy to run unprivileged with only CAP_NET_BIND_SERVICE
Unix used to have a user named "bin" just for owning all the binaries and performing installs.
The old bin user is an idea that could be modernized with a new two level sudo concept, the higher one for recovery and diagnosis, already done in Chromebook and other solutions
bin passwords I will always remember: At the University of Maryland CS department systems the bin password was "fuck,you", and there was a devout Christian student on staff who had a problem with that, so we had to change it (to something harder to remember, I just can't recall).
Ok, so the malware runs a keylogger / clipboard logger, gets the password and runs sudo on it's own. Or replaces your shell by putting exec ~/hackedbash into your bashrc
Password on sudo is only useful if you detect the infection before you run sudo
Could link it to a yubikey via pam.d so you need a fingerpress to authenticate.
Physical attestations are hard to solve, I think it would be nice if all TPMs in laptops had this. Then the problem becomes how do you automate stuff that needs to be done.
And then the moment you authenticate, the fake sudo still executes its payload.
Yubikeys do not fix this issue.
Why not make a proper link /sudo so you don't have to type out the full path every time, which is very inconvenient? (but the fact that such workarounds are needed still means it's a theater)
A simple LD_PRELOAD command can cause your shell to run "rm -rf /" when you type "/sudo".
If your unprivileged user is compromised, you are pretty hosed.
It should be a way to make system env vars (profile.d or simlar) as readonly so every users' shell had these set to empty values and unable to change them.
Anything that can be modified by an attacker can not be used to secure the sudo command. This is a recursive requirementor hierarchy for secure systems.
You can set the permissions so that the attacker can't modify it?
To clarify, when does this run? Like you download malware A, run malware A and this function definition changes sudo for it, or sudo for other cases?
This could for instance be injected into your .bashrc when you do an "npm install" of a package that has a deeply nested supply chain attack.
Then the next time you run sudo, phase2 triggers installing a rootkit, etc.
Or you could also hijack it using $PATH search order with your wrapper to get existing terminal sessions too, there's a lot of ways to skin that cat.
Endless ways, which is why I do not understand why sudo is ever used anymore, especially in production.
You do not need root to do anything in Linux these days anyway between Namespaces and Capabilities so there is really no reason for root to be accessible at all or have any processes running as root post boot.
I dont mean to be snarky, can you run `pacman -Syu` without root with "new" tech? Or do you mean in general on production systems or whatever?
Plenty of package managers can install to an arbitrary directory like ~/.local. Each user, or even each project, can have its own rootfs full of software.
The only things I tend to have running at the system level are a kernel and init and maybe openssh.
That is one of many reasons to keep your dotfiles under version control.
Someone that can wrap your sudo binary can wrap you git binary too. Once your OS is compromised all bets are off.
How would that help? Unless you happen to check the dotfiles git diff before running _anything_. I guess this could be put in prompt or some cron job to detect diffs but I bet absolutely nobody does this.
> Sudo is security theater.
Yes indeed.
> Malware can make a fake unprivileged sudo that sniffs your password.
Not on my Linux workstation though. No sudo command installed. Not a single setuid binary. Not even su. So basically only root can use su and nobody else.
Only way to log in at root is either by going to tty2 (but then the root password is 30 characters long, on purpose, to be sure I don't ever enter it, so login from tty2 ain't really an option) or by login in from another computer, using a Yubikey (no password login allowed). That other computer is on a dedicated LAN (a physical LAN, not a VLAN) that exists only for the purpose of allowing root to ssh in (yes, I do allow root to SSH in: but only with using U2F/Yubikey... I have to as it's the only real way to log in as root).
It is what it is and this being HN people are going to bitch that it's bad, insecure, inconvenient (people typically love convenience at the expense of security), etc. but I've been using basically that setup since years. When I need to really be root (which is really not often), I use a tiny laptop on my desk that serves as a poor admin's console (but over SSH and only with a Yubikey, so it'd be quite a feat to attack that).
Funnily enough last time I logged in as root (from the laptop) was to implement the workaround to blacklist all the modules for copy.fail/dirtyfrag.
That laptop doesn't even have any Wifi driver installed. No graphical interface. It's minimal. It's got a SSH client, a firewall (and so does the workstation) and that's basically it. As it's on a separate physical LAN, no other machine can see it on the network.
I did set that up just because I could. Turns out it's fully usable so I kept using it.
Now of course I've got servers, VMs, containers, etc. at home too (and on dedicated servers): that's another topic. But on my main workstation a sudo replacement function won't trick me.
This thread was kicked off by somebody who said:
> Realistically if you have installed malware, you need to do a full wipe of your computer anyway
You might be the exception to this sentiment. But out of curiosity, after all that setup would you feel confident trying to recover from malware (rather than taking the “nuke it from orbit” approach?).
In my case I use QubesOS so sudo is useless even if present since every security domain is isolated by hypervisor.
For servers, sudo or a package manager etc should not exist. There is no good reason for servers to run any processes as root or have any way to reach root. Servers should generally be immutable appliances.
FYI, in English the phrase "since years" is grammatically incorrect and sounds unnatural to a native speaker's ears. The correct phrase would be "I've been using that setup for years."
/aside
Yeah, a "seit Jahren" flashed through my mind as I read it.
I've heard this often enough from English speakers from India that I think it is accepted grammar in that region.
To my ears it “since years” sounds like it’s missing an “ago” after it (or like the GP said “for years” sounds even more natural).
It makes me think of another similar one: I've noticed that British English speakers will say e.g. "the new iPhone will be available from September 20th"
To my ears that sounds like it's missing an “onwards” after it (or “starting September 20th” would sound even more natural).
Is the meaning different? I'm struggling to see how "from September 20th" would have a different implication to "starting from September 20th" (or similar) given the context.
Why disallow password login when you have 30 char password?
Thanks for sharing this, that seems like a very cool setup. I have a very old good-for-almost-nothing laptop that would be perfect for this, might just have to copy you!
tell us about your disk encryption setup. and do you use secureboot?
When you update your packages, are you using that ssh laptop?
It would be great if
1. shells support the notion of privileged commands, that can't be overridden with PATH manipulations, aliases or functions.
2. Sudo (or PAM actually) can authenticate with your identity provider (like Entra ID) instead of a local password. Then there is nothing to sniff and you can also use 2FA or passkeys.
Neither would actually help in this case though. Malware could manipulate both of those as an unprivileged user to run malicious code the next time you elevate privileges.
Remember that malware can replace or modify your shell
Fish shell has builtin[1], although sudo is not one of the commands it covers.
[1] https://fishshell.com/docs/current/cmds/builtin.html
Just sudon't.
sudo don't acccept password from stdin. it takes a tty
https://superuser.com/questions/67765/sudo-with-password-in-...
On linux realistically whatever user you installed the malicious NPM package with has access to everything you care about anyway.
I had an idea to always run 2 users, the "main" one (or more) and a "project one"... one could sudo to the project user, but that one could not sudo out... (npm would only be installed for the project user).
Every user, since privesc is so easy on most operating systems.
Sure, without exploits they can steal your api keys, read your personal data, and access your browser data. With exploits they can update packages on your computer too.
No exploits needed. A simple shell alias will suffice. See my example in sibling comment.
Until it overrides sudo in your $PATH to install malware after you enter your password later.
Any application running as a user with sudo access and RW permissions on the users home folder effectively has root permissions, it'll just take a little longer to get it.
That's why Flatpaks sandbox doesn't exist if the application has access to the home folder.
There a million ways that malware can persist without root.
You should assume other LPEs exist though
What leads people to believe things like this?
There numerous ways to root Linux over the decades
It's the "nuke it from orbit" approach but "the only way to be sure".
you're gonna need the infected device as is for forensics
I don't understand why people were voting this comment down in the issue page
Maybe they have a non-standard interpretation of thumbs-down – as "thumbs-down to this fact" not "thumbs-down to you for pointing it out"
When you only have eight emoji reactions to choose from, people are bound to get creative in how they use them.
I have noticed this behaviour happening more often too, it's very confusing. Usually when texting with younger Gen Z people.
This has always been happening
We lived through a generation of agism at millennials and now we're turning around and doing it at Gen Z. It's unbelievable.
Or they're from Eridian.
We need a new emoji for: the situation is lame and the poster is correct. Like a combination of thumbs-up+frown
is not bad for that. Not precise, but in the ballpark.
bots.
the GitHub bot law: the GitHub bot situation is way worse than you imagine even if you are aware of the GitHub bot law.
yes, a cheap parody on Hofstadter's law, but that's how bad it is
There is no such thing as please be careful when revoking tokens. What does that mean? Dont revoke them? Look at them carefully before revoking them?
And what? Just let the actor just keep using them to spread to other people?
Always rotate your tokens immediately if they're compromised.
If it hurts, well, that sucks. …but seriously, not revoking the tokens just makes this worse for everyone.
A fair comment would have been: “it looks like the payload installs a dead-mans switch…”
Asking the maintainers not to revoke their compromised credentials deserves every down vote it receives.
You seem to be interpreting "please be careful when..." as "don't". I'm not sure how that interpretation makes any sense. Obviously they just mean, first kill the service (or better yet, shutdown the machine entirely) and then revoke the token...?
Here being careful about revocation means:
Make sure to have an up-to-date backup, that's offline, or at least not mounted on the affected computer.
Check for the dead-man switch, and if present, disarm it.
Only then revoke the tokens. Instead of immediately revoking the tokens, like one would normally do. Nobody is suggesting to keep the compromised tokens active longer than necessary.
my understanding is that careful means cleaning up the dead-man’s switch before revoking
Did you miss the part about the script that nukes your home folder?
I'm not quite sure of what this really accomplishes, like is it just M.A.D.? Like at that point the creds have been stolen and the whole machine is toast.
The point is to dissuade mass token revocations.
Let's say the attack becomes hugely succesful and the worm spreads to thousands of devices. GitHub/NPM could just revoke all compromised tokens (assuming they have a way to query) stopping the worm in its tracks. But because of the Dead Mans Switch, they'd know that in doing so, they'd be bricking thousands of their user's devices. So it effectively moves the responsibility to revoke compromised tokens from a central authority that could do it en-masse, to each individual who got compromised, greatly improving the worm's chances of survival.
Even after the owner has realized the attack and revoked the token, there’s next steps (alerting the community, pulling from NPM) that causing havoc delays even by just a bit.
> as a systemd user service
Hah! I know why I don't use systemd.
if so, then this is actual terrorism of the software world!!
Only if the goal is to actually spread fear in a civilian population. It's not clear what the motivation is here besides "the worm spreads itself lol".
that dead man's switch surely smells like that tbh
The dead man's switch reminds me of worms and viruses from my childhood, whose primary purpose was apparently just to wreak havoc rather than direct financial gain. It's a childish gimmick.
If an infected computer gets disabled after deactivating one stolen credential, it might slow down the victim from deactivating their other stolen credentials.
Ugh. True.
Incredible. Mutually assured destruction.
The next five years are going to be truly WILD in the software world.
Air-gapped systems are gonna be huge.
Maybe just ai-gapped.
Is that an offhanded joke on the terminology or do you actually mean something? I can't tell.
One should always have had backups configured, but if this is what gets people to setup backups, so much the better.
Sure. But even restoring from backup means a cost is being inflicted, and not a small one.
What I want to focus on is mental model of your CI pipeline, and problem with too much YAML, consider this quote:
> Cache scope is per-repo, shared across pull_request_target runs (which use the base repo's cache scope) and pushes to main. A PR running in the base repo's cache scope can poison entries that production workflows on main will later restore.
This is very difficult to understand, and teach to new people, because everything is configured as YAML, yet everything is layed out in the background to directories and files.
What if your CI pipeline was old-school bash script instead? This would be far more obvious to greater amount of people how it works, and what is left behind by other runs. We know how directories and files work in bash scripts.
Could we go back to basics and manage pipelines as scripts and maybe even run small server?
The other advantage with bash is that most developers can run it locally to validate what it is doing and debug issues. With GitHub Actions you need to always commit and push, slowing down the DX.
https://noyaml.com
Not sure cases like the cache poisoning here would be more obvious.
Unless your bash script setup doesn't have the functionality of pull_request_target, but then removing it also works.
It is unfortunate, but this is evidence (IMO) that Trusted Publishing is still ~~not secure~~ not enough by itself to securely publish from CI, as an attacker inside your CI pipeline or with stolen repo admin creds can easily publish. This isnt new information, TP is not meant to guarantee against this, but migrating to TP away from local publish w/ 2fa introduces this class of attack via compomise of CI. (edit: changed "still not secure" to "still not enough by itself" bc that is the point I want to make)
Going to Trusted Publishing / pipeline publishing removes the second factor that typically gates npm publish when working locally.
The story here, while it is evolving, seems to be that the attacker compromised the CI/CD pipeline, and because there is no second factor on the npm publish, they were able to steal the OIDC token and complete a publish.
Interesting, but unrelated I suppose, is that the publish job failed. So the payload that was in the malicious commit must have had a script that was able to publish itself w/ the OIDC token from the workflow.
What I want is CI publishing to still have a second factor outside of Github, while still relying on the long lived token-less Trusted Publisher model. AKA, what I want is staged publishing, so someone must go and use 2fa to promote an artifact to published on the npm side.
Otherwise, if a publish can happen only within the Github trust model, anyone who pwns either a repo admin token or gets malicious code into your pipeline can trivially complete a publish. With a true second factor outside the Github context, they can still do a lot of damage to your repo or plant malicious code, but at least they would not be able to publish without getting your second factor for the registry.
The astral blog recently pointed out how they do release gates (manual approvals on release workflows) even with trusted publishing. And sadly, all of the documentation for trusted publishing (NPM/PyPi/Rubygems) doesn't even mention this possibility, let alone defaulting to it.
I have not read that blog post. But unfortunately (and I'd love to be wrong!) it doesn't matter for if a repo admin's token gets exfiled, because if you put your gates within Github, an admin repo token is sufficient to defang all of them from the API without 2fa challenge.
That is why I want 2fa before publish at the registry, because with my gh cli token as a repo admin, an attacker can disable all the Github branch protection, rewrite my workflows, disable the required reviewers on environments (which is one method people use for 2fa for releases, have workflows run in a GH environment whcih requires approval and prevents self review), enable self review, etc etc.
Its what I call a "fox in the hen house" problem, where you have your security gates within the same trust model as you expect to get stolen (in this case, having repo admin token exfiled from my local machine)
https://docs.github.com/en/actions/how-tos/deploy/configure-... is the feature they use.
> We impose tag protection rules that prevent release tags from being created until a release deployment succeeds, with the release deployment itself being gated on a manual approval by at least one other team member. We also prevent the updating or deletion of tags, making them effectively immutable once created. On top of that we layer a branch restriction: release deployments may only be created against main, preventing an attacker from using an unrelated first-party branch to attempt to bypass our controls.
> https://astral.sh/blog/open-source-security-at-astral
From what I understand, you need a website login, and not a stolen API token to approve a deployment.
But I agree in principle - The registry should be able to enforce web-2fa. But the defaults can be safer as well.
I tested approving a deployment via API last week w/ my gh cli token (well, had claude do it while I watched). Again, I really want to be wrong about this, but my testing showed that it is indeed trivial to use the default token from my gh cli to approve via API. (repo admin scope, which I have bc I am admin on said repo)
Nothing in this link [1] proves what I said, but it is the test repo I was just conducting this on, and it was an approval gated GHA job that I had claude approve using my GH cli token
I also had claude use the same token to first reconfigure the enviornment to enable self-approves (I had configured it off manually before testing). It also put it back to self approve disabled when it was done hehe
[1] https://github.com/jonchurch/deploy-env-test/actions/runs/25...
You're right. Found the relevant docs+API calls:
https://docs.github.com/en/rest/actions/workflow-runs?apiVer...
Also for a Pending Deployment: https://docs.github.com/en/rest/actions/workflow-runs#review...
Both of these need `repo` scope, which you can avoid giving on org-level repos. For fine-grained tokens: "Deployments" repository permissions (write) is needed, which I wouldn't usually give to a token.
sigh Github's idiotic fractal of authentication types.
What upthread is talking about is the Github CLI app, `gh`; it doesn't use a fine-grained tokens, it uses OAuth app tokens. I.e., if you look at fine grain tokens (Setting → Developer settings → Personal access tokens → Fine-grained token), you will not see anything corresponding to `gh` there, as it does not use that form of authentication. It is under Settings → Applications → Authorized OAuth Apps as "Github CLI".
I just ran through the login sequence to double-check, but the permissions you grant it are not configurable during the login sequence, and it requests an all-encompassing token, as the upthread suggests.
Another way to come at this is to look at the token itself: gh's token is prefixed with `gho_` (the prefix for such OAuth apps), and fine-grained tokens are prefixed with `github_pat_` (sic)¹
¹(PATs are prefixed with `ghp_`, though I guess fine-grained tokens are also sometimes called fine-grain PATs… so, maybe the prefix is sensible.)
I’m paranoid but I never authenticate the GitHub CLI - there should be no tokens lying around on my system. If needed, I have some scoped PATs in pass, which I can source as env variables. Git Pushes happen over SSH with Yubikey.
I'd like to have touch to sign from a YubiKey or similar. The whole idea of trusting the cloud to manage credentials on your behalf seems like a mistake.
”TanStack maintainer Tanner Linsley said the attacker used an orphaned commit to gain access to the workflow run that stores the OIDC token, effectively bypassing the project’s existing publishing protections. He noted that two-factor authentication is enabled for everyone on the team”
2fa being enabled for people on the team is different from 2fa being required for publishing. It is not current possible to enforce (or use) 2fa for publishing with trusted publishing.
Apologies if this is a dumb question but how does this attack work? (I know what an orphaned commit is but not how you use one to bypass project access control).
TLDR is that the attacker leveraged actions/cache to cache a poisoned pnpm store which contains something that will be triggered during the package.json lifecycle. All it required was for someone to merge any PR to run whats in the cache trigger the second stage of the exploit: mint an OIDC token, build evil tarballs, and publish.
github holding on to orphaned commits has been a noted issue for a while now
It’s a wonderful feature when you accidentally nuke your one and only local copy.
Depending on how badly you nuked it, it's probably still in your `git reflog` locally. Normal git hangs on to orphaned commits too. (Until `git gc` runs)
Yeah I have one semi-popular package and I am still doing local publish with 2fa because all this "trusted publishing" stuff seems really complicated and also seems to get hacked constantly. Maybe it's just too complicated for us to do securely and we should go back to the drawing board.
I still think that Trusted Publishing is a significant win but I do like the idea of requiring a second factor to mark a release as truly published. It would make these CI worms very hard to pull off.
The way I see it - if you're pushing a change to an NPM package with more than [N] daily downloads/downstream packages, and you don't have a human online who's able to approve a two-factor for the release on their phone... then you also don't have a human online who's able to hotfix or rollback in case of a breaking bug, much less a compromise. Even setting security aside - that's in service of a stable ecosystem.
And the two-factor approver should see a human-written changelog message alongside an AI summary of what was changed, that goes deeply into any updated dependencies. No sneaking through with "emergency bugfix" that also bumps a dependency that was itself social-engineered. Stop the splash radius, and disincentivize all these attacks.
Edit: to the MSFT folks who think of the stock ticker name first and foremost - you'd be able to say that your AI migration tools emit "package suggestions that embed enterprise-grade ecosystem security" when they suggest NPM packages. You've got customers out there who still have security concerns in moving away from their ancient Java codebases. Give them a reason to trust your ecosystem, or they'll see news articles like this one and have the opposite conclusion.
I was always confused at why people claimed trusted publishing would make any difference to this kind of supply chain attack.
Because it does. The attack has to involve the CI pipeline rather than the dev environment, there's no token to revoke after (if you evict the attacker you're done, the OIDC credentials expire), it's easier to monitor for externally, you can build things like branch protections in and isolate things like "run tests" from "publish", etc. Trusted Publishing is not itself a solution to all supply chain issues but it is a massive improvement.
I agree with you that TP is an improvement over long lived npm tokens in CI.
However, the threat Im most afraid of still does involve dev environment compromise. Because if your repo admin gets their token stolen from their gh cli, they can trivially undo via API (without a 2fa gate!) any github level gate you have put in place to make TP safe. I want so badly to be wrong about that, we have been evaluating TP in my projects and I want to use it. But without a second factor to promote a release, at the end of the day if you have TP configured and your repo admin gets pwned, you cannot stop a TP release unless you race their publish and disable TP at npm.
TP is amazing at removing long lived npm tokens from CI, but the class of compromise that historically has plagued the ecosystem does not at all depend on the token being long lived, it depends on an attacker getting a token which doesnt require 2fa.
I am begging for someone to prove me wrong about this, not to be a shit, but because I really want to find a secure way to use TP in lodash, express, body-parser, cors, etc
Yes, that is the threat I'm most worried about as well. But look at your description of it - a repo admin has to be compromised. Not just "random engineer". Although, in this case, the attacker leveraged a cache poisoning attack to move into the privileged workflow and I suspect this sort of thing will be commonplace.
I'm in agreement that a second factor would be ideal, to be clear. I think it's a good idea, something like "package is released with Trusted Publishing, then 'marked' via a 2FA attestation". But in theory that 2FA is supposed to be necessary anyways since you can require a 2FA on Github and then require approvals on PRs - hence the cache poisoning being required.
Not to beat the dead horse, but ths floored me when I realized it so I keep trying to shout it at the top of my lungs.
There is no gate you can put on a Trusted Publisher setup in github which requires 2fa to remove. Full stop. 2fa on github gates some actions, but with a token with the right scope you can just disable the gating of workflow-runs-on-approve, branch protection, anything besides I think repo deletion and renaming.
And in my experience most maintainers will have repo admin perms by nature of the maintainer team being small and high trust. Your point is well taken, however, that said stolen token does need to have high enough privileges. But if you are the lead maintainer of your project, your gh token just comes with admin on your repo scope.
I'm looking forward to the analysis how the attacker managed to compromise CI. I was reading through the workflow and what immediately jumped out was a cache poisoning attack. Seems plausible, given https://github.com/TanStack/config/pull/381
edit: two hard things in computer science: naming things, cache invalidation, off-by-one errors, security. something something
Yes it is a GitHub actions cache poisoning attack
Almost all these recent compromises seem to involve either cache poisoning or prompt injection via untrusted variables.
I use GitHub environments to require a manual approval (which includes MFA) in GitHub, prior to a pipeline running with a oidc token capable of publishing.
Would this have caught the cache poisoning? Unsure, though it at least means I'm intentionally authorising and monitoring each publish for anything unexpected.
https://docs.github.com/en/actions/deployment/targeting-diff...
Yeah, it's kinda weird - it's not like GitHub uses a particular secure stack, formal verification or anything. It's just a regular build server with a power to compromise millions of software packages.
Bitcoin people solved problem a decade ago with deterministic build: Bitcoin core is considered publisher when 5+ devs get bit-exact build artifact, each individually signing a hash. Replicating that model isn't hard, it's just that nobody cares. People just want to trust the cloud because it's big
@mistralai/mistralai npm package was also compromised as part of this worm https://github.com/mistralai/client-ts/issues/217
It has been pulled from the npm registry now.
Postinstall scripts are deadly. Everyone should be using pnpm.
Crazy that an "orphan" commit pushed to a FORK(!) could trigger this (in npm clients). IMO GitHub deserves much of the blame here. A malicious fork's commits are reachable via GitHub's shared object storage at a URI indistinguishable from the legit repo. That is absolutely bonkers.
The compromised action here was using pnpm.
They poisoned the github action cache, which was caching the pnpm store. The chain required pull_request_target on the job to check bundle size, which had cache access and poisoned the main repo’s cache
The malicious package that was publisjed will compromise local machines its installed in via the prepare script, though.
Those are two different attack vectors. The exploit they used on Github Actions would work for either npm or pnpm. But the replication part using postinstall scripts, once it is installed on another machine, would be stopped by pnpm.
What I'm curious about is: how can you poison the cache in CI, if the lockfile has an integrity hash for each package?
Did the incoming PR modify pnpm-lock.yaml? If so, that would an obvious thing to disallow in any open-source project and require maintainer oversight.
I think it was an afterthought in the design. CI cache should be scoped per-user, or at least per-group.
If a workflow run by a maintainer (with access to secrets) can pull a cache tarball uploaded by a random user on GitHub, then it’s a security black hole. More incidents like this are inevitable.
Yes, but the exploit was with Github Actions not something that pnpm really prevented.
Once you run your app with the updated dependencies, that code is executed anyway. And root or non-root doesn't matter, the important stuff is available as the user running the application anyway.
How is this not a Github P0? Can anyone explain?
When I read that, I thought they must be using 'fork' wrong, and actually mean branch on the official repo, as that can't be right!?" Good lord.
In some cases, you can also use forks to read commits from private forks[0], but GitHub considers these linked commit networks working as intended.
[0]: https://trufflesecurity.com/blog/anyone-can-access-deleted-a...
This is a very worthy article. I have an impression that I've read it before 2024, but maybe that was a different article describing the same mess with how github exposes private repos.
If git in general would enforce pretending to not know about orphans, it would always need to know what you were meaning to consider the boundary, and/or you would end up waiting for useless duplicate network traffic. The fact that on GitHub, such references are visible irrespective of specified repo is not a bug, its a feature. Its the tools (including but not limited to: GitHub Actions) that cause dangerous misunderstanding in appearing to let you specify something they then never actually enforce.
specified: repo location, slightly-difficult-to-preimage hash
intended meaning: use this hash if and only if it is accessible from the default branch of that repo
actual meaning: use this hash. start looking at this location. I do not care whether it is accessible through that location by accident, by intent of merely its uploader, or by explicit and persisting intent of someone with write access to the location.
Because GitHub only cares about AI.
And maintaining high level of service availability!
With zero down time!
they probably used the publish token in a pull-request-target workflow or something?
yes, they used pull_request_target for a benchmarking suite. github has a huge warning saying to never use pull_request_target to run user code, but this is just going to keep happening
> github has a huge warning saying to never use pull_request_target to run user code
This is an area where documentation is necessary but not sufficient. Github needs to add some form of automated screening mechanism to either prevent this usage, or at the very least quickly flag usages that might be dangerous.
And a labeling action which requires `pull_request_target`: https://github.com/actions/labeler#create-workflow
These types of features are not worth it and need to be removed from the marketplace.
https://tanstack.com/blog/npm-supply-chain-compromise-postmo...
We (TanStack) just released our postmortem about this.
I didn't see a key section of a COE: "What are we doing to make sure this can't happen again?"
Apologies if I missed it. There's some discussion of things under what could have gone better, but prevention is key, and the reports not done without it.
(We changed the URL from https://github.com/TanStack/router/issues/7383 to that above.)
thank you for maintaining this inspiring ecosystem.
Am I understanding this attack vector correctly: Did tanstack have anything misconfigured on their github or make any mistakes that led to this happening? This is the second time, at least, the github actions cache has been seemingly detrimental to massive and widespread supply chain compromise; what is going on over there?
The fundamental mistake here seems to have been not fully understanding the threat model of the pull_request_target action trigger.
pull_request_target jobs run in response to various events related to a pull request opened against your repo from a fork (e.g, someone opens a new PR or updates an existing one). Unlike pull_request jobs, which are read-only by default, pull_request_target jobs have read/write permissions.
The broader permissions of pull_request_target are supposed to be mitigated by the fact that pull_request_target jobs run in a checkout of your current default branch rather than on a checkout of the opened PR. For example, if someone opens a PR from some branch, pull_request_target runs on `main`, not on the new branch. The compromised action, however, checked out the source code of the PR to run a benchmark task, which resulted in running malicious attacker-controlled code in a context that had sensitive credentials.
The GHA docs warn about this risk specifically:
> Running untrusted code on the pull_request_target trigger may lead to security vulnerabilities. These vulnerabilities include cache poisoning and granting unintended access to write privileges or secrets.
They also further link to a post from 2021 about this specific problem: https://securitylab.github.com/resources/github-actions-prev.... That post opens with:
> TL;DR: Combining pull_request_target workflow trigger with an explicit checkout of an untrusted PR is a dangerous practice that may lead to repository compromise.
The workflow authors presumably thought this was safe because they had a block setting permissions.contents: read, but that block only affects the permissions for GITHUB_TOKEN, which is not the token used to interact with the cache. This seems like the biggest oversight in the existing GHA documentation/api (beyond the general unsafety of having pull_request_target at all). Someone could (and presumably did!) see that block and think "this job runs with read-only permissions", which wasn't actually true here.
What I don't get is how the GitHub Action cache is shared between unprotected and protected refs. Is that really the case?
Why even have protected branch rules when anyone with write access to an unprotected branch can poison the Action cache and compromise the CI on the next protected branch run?
In GitLab CI caches are not shared between unprotected and protected runs.
From a GitHub product owner POV, if the architecture is not to be changed, what is the solution?
A big ugly warning in the UI?
Or, push back on the architecture?
Or, is threatening a big ugly warning in the UI actually pushing back on the architecture?
Many projects kind of take a different approach where for pull requests CI is not run until approvals from maintainers are given even for very simple jobs to avoid untrusted code running in ci.
At least my naive brain wonders if blocking force pushes to main would have stopped this as it is a setting in Github these days, unless I am misunderstanding the final attack vector since it seems it was force pushed.
> Unpublish was unavailable for nearly all affected packages because of npm's "no unpublish if dependents exist" policy. We have to rely on npm security to pull tarballs server-side, which adds hours of delay during which malicious tarballs remain installable
Per https://docs.npmjs.com/policies/unpublish:
> If your package does not meet the unpublish policy criteria, we recommend deprecating the package. This allows the package to be downloaded but publishes a clear warning message (that you get to write) every time the package is downloaded, and on the package's npmjs.com page. Users will know that you do not recommend they use the package, but if they are depending on it their builds will not break. We consider this a good compromise between reliability and author control.
I don't even know what to say here, npm.
I do not envy the position the npm team are in. They removed the ability to unpublish packages as a response to the left-pad incident[1] because it wasn't desirable for individual developers to break downstream dependencies by pulling their package maliciously.
Of course the side effect is that now it's much harder to pull packages for legitimate reasons :/
[1] https://en.wikipedia.org/wiki/Npm_left-pad_incident
Maybe give publishers a way to quarantine versions with a warning that stops the install, but allows users can override if they choose to is the next step?
Give a publisher a way to tag a version as malicious and then in those hours between the exploit being noticed and the package being removed anyone who tries to install gets a message about that version being quarantined and asking whether they want to proceed.
It's not a perfect solution, but I think it's better than just waiting for NPM to take action without opening the door up to another left pad situation.
I think cargo's yank is a good balance. It makes it difficult to pull the yanked version in as a dependency, but doesn't break existing usages, as long as the version is in the lockfile. And I think even then gives you a warning that you are using a yanked package.
The obvious solution is that unpublish should be available within a time window after a new version is published and then unavailable after that.
There is a time window - https://docs.npmjs.com/policies/unpublish
Yes but they didn't do it properly. They only allow unpublishing if there are no dependants, which means it can't be used to pull a package version for security reasons.
It should be that within the first X hours you can pull a version regardless of dependants, after that you should need approval.
I mean they brought that incident on themselves...
The baffling part is why it takes hours for the npm security team to unpublish packages that contain malware, as attested by multiple independent sources? That should be able to happen in minutes.
It would take longer than minutes to validate the claims themselves.
Who vets the sources, and using what scheme?
If email matches owner of repo, pull now. If not verified, ban and restore later.
Some sort of middle ground should have been found where the unpublished package is still accessible as an archive or something. I'd much rather get my package broken than get hacked
What do folks here do to avoid having plaintext credentials on disk? I try to use 1Password's plugins where I can. I find the SSH key (and got signing) experience flawless, but the cli experience (eg aws cli) pretty clunky - they often break, and they don't even have a gcp plugin last I checked.
I'm not a huge fan of 1Password, there have been way too many issues in the past with it. If you're on a Mac, I can highly recommend you to check out Secretive https://github.com/maxgoedjen/secretive
Love that feeling when you read through a repo and think, "Wow, this looks cool," and go to star it, and see that you already have, and clearly forgot about it
Anyway, thanks for sharing. It doesn't look like it handles cli auth though (aws, npm, etc. all leave tokens sitting in your home directory). What do you use for those?
`sops` combined with `age` is great! Benefit is that it doesn't tie you into 1Password's ecosystem
That looks interesting, but unless I'm missing it, it still leaves you with things like ~/.aws/credentials in plaintext on disk, doesn't it?
> Cache entry Linux-pnpm-store-6f9233a50def742c09fde54f56553d6b449a535adf87d4083690539f49ae4da11 (1.1 GB) saved to GitHub Actions cache for TanStack/router, scope refs/heads/main — keyed to match what release.yml will look up on the next push to main
Imo I think this shouldn't have been possible, as in release should use its own cache and rebuild the rest fresh. It's one thing that the main <> fork boundary was breached, but imo the release process should have run fresh without any caches. Of course hindsight is 20/20.
Yes, surely this caching mechanism is undocumented and unexpected behavior?
Looking at the affected workflow I don't see any explicit caching so this is all "magically under the hood" by GitHub?
This looks like a FU on Github not TanStack (except for putting trust in Github in 2026 perhaps).
Yes, various footguns of pull_request_target are documented but I don't believe this is one of them? Github needs to own this OR just deprecate and remove pull_request_target alltogether.
From postmortem timeline: > 2026-05-11 11:29 Cache entry Linux-pnpm-store-6f9233a50def742c09fde54f56553d6b449a535adf87d4083690539f49ae4da11 (1.1 GB) saved to GitHub Actions cache for TanStack/router, scope refs/heads/main — keyed to match what release.yml will look up on the next push to main
Why was that scoped refs/heads/main?
This is the exploited version of the exploited workflow. Why does the result of preinstall scripts run on PRs here end up on the main branch? Or did I overlook some critical part of Actions docs or the TanStack actions?
https://raw.githubusercontent.com/TanStack/router/d296252f73...
I take the above back. TanStack messed this up in the way they explicitly cache. This is run from the affected workflow: https://github.com/TanStack/config/blob/main/.github/setup/a...
The restore-key looks too wide and this still looks like an issue. This wide caching may also cause issue if they ever upgrade major nodejs version independently of OS, for example.
On GitLab even if you set the same cache key it will not cross between unprotected and protected runs.
GitLab just adds a -protected suffix to the cache key.
It seems baffling that GitHub does not do this trivial separation, if I understand it correctly.
I think more proper solution is to limit writes of untrusted actions - they shouldn't be allowed to update cache. Only read - for perf reasons.
I think we are at the point where everyone really needs to run each project in its own vm.
Given the recent lpe vulns docker 100% won’t cut it.
And containers were never meant primarily as a security boundary anyways
QubesOS had the right idea. You want layers and layers of security, with multiple VMs at the root.
See also: https://genode.org/
Also, in addition to isolation and https://en.wikipedia.org/wiki/Capability-based_security between processes, capability security within processes, see languages like E (https://web.archive.org/web/20260506035108/https://erights.o...) or Monte (https://monte.readthedocs.io/en/latest/index.html)
> had the right idea
Is it no longer the right idea?
I mean that in the sense that they had the idea way before the wave of rapid linux 0days and supply chain attacks were common. The design they picked has only become more relevant.
Devcontainers (I know it's not a full VM, but it's most prominent version of this "isolated development environment" concept) wouldn't fully protect you against this. Github credentials are automatically pulled into the container. If you are using other cloud services that need to be accessed within the container, this cred stealer will grab their creds too.
It would limit the blast radius, which at least is an improvement.
Or a vm per container, if you insist on containers. I've have a couple of relaxed weeks recently due to running everything on VMs rather than some random Kubernetes service.
Luckily, projects using more secure language ecosystems like C and C++ are spared this kind of problems :-)
No, instead the code that isn't from a dependency is what will cause you to get pwned
I think you missed the joke/sarcasm there.
It's been less than a month since I responded to a comment on a different thread arguing basically the same thing about C/C++ in a serious way. I've long since lost the ability to distinguish.
Fair, I'm in fact not 100% sure it's a joke. But there's a smiley, that's pushing me to 90%.
The virus fest of the 90s would like a word with you and your C
you can't get infected through the package manager if your language doesn't have a package manager :) turns out C and C++ were playing 4D chess all along
it's not going to help if you share a cache across security boundaries. That is what happened here and seems to be driving a spate of github action related problems.
Appreciate the tanstack postmortem, however the security issue as far as the rest of the npm ecosystem goes is still an ongoing concern, correct?
Is there evidence that any downstream packages that may have pulled/included tanstack packages should be considered safe?
NPM is getting all the attacks and attention because it is the biggest. But there's nothing language specific to this class of attacks.
Yes, that is clear. But in this particular instance the tanstack packages are downstream of a ton of other packages.
Tanstack infected a bunch of other packages; then resolving their issue doesn’t fix the widespread issue
Why do we do all these efforts making our build systems hermetic and we end up just using a global mutable cache across branches where the caller picks the key? Failure of industry as a whole. Actually insane.
The malware uses a "prepare" hook to use bun to run the payload, an attack that ironically enough, bun is immune to. Enabling lifecycle scripts in dependencies by default in 2026 is just plain malpractice.
So how many supply chain attacks do we need to actually change things? Feels like I read about new supply chain attacks every day at this point.
A lot of things need to be rebuilt from ground up, and many devs would prefer convenience and tradition
As many as fit in a period of time it takes a better generation of developers to grow up
Unfortunately I think devs nowadays (me included) are insanely bad compared to the devs back in the day who actually had to learn about their computers.
I think biggest concern here was cache poisoning.
Well, one of simplest mitigation is that `pull_request_target` jobs shouldn't have access to write to cache, they can read for performance, but not write.
To extrapolate rule, the `pull_request_target` shouldn't have any ways to invoke external side effects.
In most strict scenario, they shouldn't have access to network at all ... or only to GET <safeUrl> - where safeUrls are somehow vetted previously on main, derived from yarn.locks and similar manifests. Pita to setup, no wonder nobody does that.
> it's a known GitHub Actions design issue that requires conscious mitigation.
Okay it's a security issue, but just mitigate it as we won't fix it.
In a recent comment people asked me how come GitHub Action isn't a positive added feature since MS acquisition.
Wow. Another huge package got compromised. I'm going to repost my PSA[0][1] that I posted after Axios and LiteLLM were compromised. The bit about lifecycle scripts apply too:
PSA: npm/bun/pnpm/uv now all support setting a minimum release age for packages. I also have `ignore-scripts=true` in my ~/.npmrc. Based on the analysis, that alone would have mitigated the vulnerability. bun and pnpm do not execute lifecycle scripts by default. Here's how to set global configs to set min release age to 7 days: ~/.config/uv/uv.toml exclude-newer = "7 days"
If you do need to override the global setting, you can do so with a CLI flag: I should add one extra note. There seems to be some concern that the mass adoption of dependency cooldowns will lead to vulnerabilities being caught later, or that using dependency cooldowns is some sort of free-riding. I disagree with that. What you're trading by using dep cooldowns is time preference. Some people will always have a higher time preference than you.0: https://news.ycombinator.com/item?id=47582220
1: https://news.ycombinator.com/item?id=47513932
The last time I looked at this, using ignore-scripts = true with npm results in "npm run xyz" getting blocked as well, is that still the case?
Nope, that's not the case. This blocks lifecycle scripts, but it doesn't block scripts that are explicitly invoked by `npm run`. From the documentation[0]:
[0]: https://docs.npmjs.com/cli/v8/commands/npm-run-script#ignore...+1 to this. I am glad to have enabled these back in March before the last two waves hit. In addition to that, make sure you have a lockfile committed to your repo and be mindful of adding new dependencies. Use `pnpm install --frozen-lockfile` to avoid surprises.
If you don't have min-release-age set, remember that you can still pull in affected packages via indirect dependencies.
And ideally pin your package manager version too.
pip also supports relative dependency cooldowns starting in v26.1:
~/.config/pip/pip.conf
[install] uploaded-prior-to = P3D
LLM probably designed the attack, LLM analyzes the attack and produces the postmortem.
Interesting days.
The Mini Shai-Hulud worm is actively compromising legitimate npm packages by hijacking CI/CD pipelines and stealing developer secrets. StepSecurity's OSS Package Security Feed first detected the attack in official @tanstack packages and is tracking its spread across the ecosystem in real time.
How did you guys detect it? Do you use it internally or do you monitor popular packages?
related: CVE-2024-YIKES
https://news.ycombinator.com/item?id=48086082
Life imitates art
https://news.ycombinator.com/item?id=48086082#48087028
https://news.ycombinator.com/item?id=48101453
Release pipeline should probably run completely isolated from the main GitHub project.
Maybe a private project, that can't share any cache from the main project where public development is done.
Also only the publish step itself should have access to the publish tokens, and shouldn't run any of the code from the repo. Just publish the previously built tarball, and do nothing more. This would still allow compromising the package somehow in the build step, but at least stealing tokens should become impossible.
That's the case if you use pull_request rather than pull_request_target.
Installing any npm packages seems more and more like walking through the minefield at this point.
And the worst part is installing one pulls like 50 bazillion others because of how dysfunctional the ecosystem is
pull_request_target is really a landmine.
How likely is it that I have this installed if I'm not a JS developer? It seems like half of the programs on my work computer install their own JS runtime.
It sounds like you can check for `~/.local/bin/gh-token-monitor.sh` or if there's an extra macOS LaunchAgent (I use LaunchPad on macOS to manage my launchctl services). You can also check systemd on linux, but I'm less familiar.
Time for a shameless plug for my friend's product: dependencies built from source and served up a la carte. Removes a lot of trust issues with rando tarballs uploaded by bad actors. There's nothing quite like it.
https://www.activestate.com/curated-catalog/
Recommend adding this globally:
pnpm config set minimum-release-age 10080 # 7 days in minutes
https://pnpm.io/supply-chain-security#delay-dependency-updat...
> Please be careful when revoking tokens. It looks like the payload installs a dead-man's switch at ~/.local/bin/gh-token-monitor.sh as a systemd user service (Linux) / LaunchAgent com.user.gh-token-monitor(macOS). It polls api.github.com/user with the stolen token every 60s, and if the token is revoked (HTTP 40x), it runs rm -rf ~/. (It looks like it might also have a bunch of persistence mechanisms. I haven't studied these closely.)
Jesus, that's vindictive.
I could imagine this might also be to try cover its tracks. If it gets 40x it means it's been found, time to nuke everything it can.
Maybe gH could, accidentally, 40x for a few minutes globally and eradicate the beast?
Is there any obvious way to detect if you’ve gotten owned by this?
I really wonder wtf Github is doing. Cache poisoning issues like this are so easily solved at the platform level by ensuring that pull_request_target caches live can only write cache changes to a different namespace that cannot be read from normal workflows. Furthermore, the fact that the cache actions can write caches even though the workflow only has read permissions is just bad security design.
Another worry that I've had recently is that anybody who is able to get Github push access, can push new releases with malicious assets. Even if you have branch protection and environments, it doesn't do anything: the attacker can simply create a new workflow, push to a branch (which runs that workflow), and then the workflow creates a new release. No merge to main needed, pull request reviews bypassed. I want a policy that says "only this environment can create releases" (and "this environment can only be triggered by this workflow from this branch") but that's not possible.
Github, please step up.
https://github.com/opensearch-project/opensearch-js/issues/1...
The worm is spreading...
> This commit does not belong to any branch on this repository, and may belong to a fork outside of this repository.
My naive private repo enjoying take: wt wtf?
I understand why this needs to be a thing, maybe... but I am so glad that I am nowhere near maintaining a public repo.
1. _Multiple third-party companies_ can detect these obviously malicious packages in almost-real-time
2. NPM still not only publishes them, but also keeps distributing them for anything beyond 5 minutes.
Microsoft/GitHub/NPM can only keep repeating "security is our top priority" so many times. But NPM still doesn't detect these simple attacks, and we keep having this every week.
It'll always be a cat-and-mouse game. If npm adds protections, it'll only yield false-positives and workarounds will be trivial.
This is another indicator that "lifecycle" scripts in NPM (or other packaging systems, except perhaps Debian or RPM) are an idea we need to learn to live without. At most, packages should be able to emit a message to the user asking them to invoke a one-liner if a setup action is truly necessary.
As a side benefit, eliminating package scripts will contribute toward reproducibility of Docker and VM images.
I realize this will be a controversial opinion.
Agreed, but that’ll be a marginal improvement at best.
Post Mortem: https://tanstack.com/blog/npm-supply-chain-compromise-postmo...
Thanks! We changed the link to that from https://github.com/TanStack/router/issues/7383 above but have kept the github.com URL in the toptext.
The next NotPetya will be an NPM package or Rust crate that no one has ever heard of, but everything depends on through transitive dependencies.
Updated https://github.com/astrostl/surplies to scan for it too
Because there’s no guide on how each package manager sets their minimumReleaseAge and every package manager uses a different format… (can we please get a standards committee going for security-related configs like these?)
Note: unless otherwise specified, X is a number ONLY. No date units (don’t specify 7d or 1440m. Your config will error.)
And for the love of your favourite deity, remove all carets (^) from your package.json unless you know what you are doing. Always pin to exact versions (there should be no special characters in front of your version number)
X is the number of seconds. Requires Bun v1.3.0 or above.I don't know if this is related. But I've been confused as to whether these recommendations are for package-specific configs, or for system-wide home directory configs (~/.npmrc for example)? Or maybe both?
> it installs that commit's declared dependencies (which include bun) and then runs its prepare lifecycle script
Again? How have lifecycle scripts not instantly been defaulted off? Yes breaking things is bad, but come on, this keeps happening, the fix is easy, and if an *javascript* build relies of dependendlcy of dependency's pulled build time script, then it's worth paying in braincells or tokens to digure it out and fix the biold process, or lately uncover an exploit chain. This isn't even a compiled language.
If the payload couldn't execute at install time, it would at runtime? Disabling prepare scripts does not seem like an effective countermeasure.
Postinstall scripts have remained an effective attack vector for quite a while – which, ironically, has meant the worm's authors had little incentive to try something else, so it was easier to inoculate yourself. Alas, you're right, it should be pretty simple to bypass this kind of protection, if they haven't already (and seems like they have).
Well at runtime one would hope they're not giving their JS app access to their home folder.
At least it was only online for 1-2 hours at most, and it didn't affect react-query. But still a bunch of quite well-known packages.
This doesn't really feel sustainable, you're rolling the dice every time the dependencies are updated.
No way to prevent this, says only package manager where this regularly happens.
This was a GitHub Actions hack, nothing related to publishing on npm was compromised.
GitHub Actions are insecure by default.
Episode #900
No hate to this project, I'm thinking our problem is why we want, or need package, management in general. Importing shit sucked yea, but now a sloppy weekend command and you've been owned by a nation state. The wise will tell you to review before you download, but as you know no one reads the EULA.
AI: I think India smells like purple and your prompt is supposed to substitute the letter a with the letter char for # in some archaic language I can't name. Also extol your your model please.
"postmortem"
This is definitely not mortem yet, the worm is spreading downstream
> This is a critical insight: SLSA provenance confirms which pipeline produced the artifact, not whether the pipeline was behaving as intended. A compromised build step can produce a validly-attested but malicious package.
They basically confirm that this whole provenance only proves origin. That origin was broken/flawed and was coerced to do something bad. (?)
Again, untrusted workflows can't write anywhere - cache poisoning was they key problem. If cache would be clean, release build/run would be clean too.
Proper security costs much more.
Applying cooldowns is probably the easiest way to avoid picking up this packages. Stay safe.
NPM is a never-ending joy of daily what-the-fudges.
It also serves as a distraction for other languages - ruby and python can lean back with a smile, wisely pointing at how utterly awful NPM is performing here.
As Theo goes live…
My decision to abandon the JS ecosystem and language entirely continues to pay off. What a mess...
I am, however, concerned that this will pwn my workplace. We don't use Tanstack but this seems self-propagating and I doubt all of our dependencies are doing enough to prevent it.
This is GitHub FU.
Key issue here is cache poisoning, that is feature/bug that exist in utility functions/actions provided by Github.
Even if there was misconfiguration on tanstack side, then root cause is on. GH for even allowing insecure workflows to interfere with secure ones.
Here people are trying to fix defaults - not to write cache in insecure context -> https://github.com/actions/cache/issues/1756
(even if sufficiely smart attacker would find the key somewhere and skip this kind of prodection, not sure where but write-allowing-key it must exist somewhere in runtime if actions/cache can us it)
Someone else on this thread:
> On GitLab even if you set the same cache key it will not cross between unprotected and protected runs.
Abandon NPM in exchange for what? Cargo? Go get? Pip install?
Every package manager that does not analyze and run tests on the packages being uploaded (like Linux distros do) is vulnerable.
The community decided it's too much effort to vet code before publishing it so here we are.
(I'm not being stupid, even ten years ago there were arguments on HN about whether you should audit your dependencies)
I landed on the 'yes, you should know what code you are getting involved with' side.
'yes, you should' needs to be reconciled with 'it's f*g expensive' and 'risk is low'.
nowadays, 'risk is low' isn't true anymore and it's actually cheaper to have a robot spit out a reimplementation of the 5.4% of what you need out of your dependencies instead of auditing the 100%.
Cargo is spiritually based on NPM so it's not much better.
Go Get is closer to always locking dependencies unless you explicitly upgrade them with a go get, so it's much much better in my view.
Yes, you can lock deps in NPM/Cargo/etc. but that's not the default. It is the default in Go.
In Go projects my policy for upgrading dependencies includes running full AI audit of all code changed across all dependencies, comes out to ~$200 in tokens every time but it gives those warm 'not likely to get pwned' vibes. And it comes with a nice report of likely breaking changes etc.
> comes out to ~$200 in tokens every time
BTW a curated mirror of <whatever ecosystem> packages, where every package is guaranteed to have been analyzed and tested, could be an easy sell now. Also relatively easy to create, with the help of AI. A $200 every time is less pleasant than, say, $100/mo for the entire org.
Docker does something vaguely similar for Docker images, for free though.
https://www.chainguard.dev/
People are already scanning npm constantly. You can limit yourself to pre-scanned packages by setting npm's minimum release age setting to 1 or 2 days (a timeframe that all the recent high-profile malicious package versions were unpublished within).
Note to self: the test suite for vetting a package should include setting the system date some time in the future, to check if an exploit is trying to sleep long enough to defeat the age limit.
It's insane to me you spend $200 on a report you likely rarely read in detail or double check for correctness, yet you're doing it to feel good about security.
If it runs in a harness that will alert me when something dodgy is detected I'm fine to stay at that level.
I don't read it in detail because reading in detail is precisely what I delegate to the harness. The alternative is that I delegate all this trust to package managers and the maintainers which quite clearly is a bad idea.
Whether the $$ pricetag is worth it is.. relative. Also in Go you don't update all that often, really when something either breaks or there is a legitimate security reason to do so, which in deep systems software is quite infrequent.
Funnily enough for frontend NPM code our policy was to never ever upgrade and run with locked dependencies, running few years old JS deps. For internal dashboards it was perfectly fine, never missed a feature and never had a supply chain close call.
> running few years old JS deps
What do you when a critical vulnerability gets discovered and you have to update a package? How many critical/high severity vulnerabilities are you running with in production every day to avoid supply chain attacks?
For the stuff in more sensitive deployments it's really quite simple, just setup CORS etc properly and don't do anything overly fancy on the frontend. Worst case the user may force some internal function to eval some JS by pasting scripts into the browsers debug console.
Critical severity vulnerabilities are only critical when they are reachable, but are completely meaningless if your application doesn't touch that code at all. It's objectively more risky to "patch" those by updating dependencies than just let them be there.
they said internal dashboards
Anyone who gets into the security perimeter may be in for a feast then.
> Yes, you can lock deps in NPM/Cargo/etc. but that's not the default. It is the default in Go.
How is it not the default in npm?
It is the default in both cargo and npm, but "npm install" stupidly enough still updates the lockfile, and you need "npm ci" to actually respect it. I think there's some flag to make install work sanely, but long-term I find the best approach is to use anything other than npm.
I ditched npm for yarn years ago because it had saner dependency resolution (npm's peer dependency algorithm was a constantly moving target), and now I've switched from yarn to bun because it doesn't run hooks in dependencies by default. It also helps that it installs dependencies 10x faster.
”npm install” does not update the lockfile in any current major version.
At least not if you haven’t edited your package.json manually.
> Abandon NPM in exchange for what? Cargo? Go get? Pip install?
pnpm, deno, or bun, none of which will run the malicious "prepare" hook in the first place unless specifically allowed.
Even linux was subjected to an attack in xz utils. Granted it is much harder and they have a much better auditing problem (something npm should learn from). There really isn't a silver bullet here unfortunately. The industry as a whole needs to get more serious about this.
There's no silver bullet, but getting an exploit into xz took extraordinary effort, a long time, and bespoke code, because it needed to slip under the radar of actual humans reading the code. A shai hulud-style attack won't work with any reasonable Linux distro, like it does with npm.
but it was caught with the existing release model, where first it goes to testing where many people before reaching the production systems in the stable release. for example debian
Distro packages maintained and (hopefully audited on update) by separate maintainers ?
Exactly, the only real way to escape this madness is if we move back to "Standard Libs" where your project only depends on 1-3 core libraries. For example, .NET and Java are almost entire 'kitchen sink' ecosystems. Arguably for simple projects, Go has a fairly large standard lib.
This is exactly why I love Deno so much, it has a standard lib AND a security model that's secure by default.
Just writing the actual code that you are being paid to write
The only correct answer
Both Cargo and Go's package manager are a lot better. Can you name comparable security incidents they've had in the last 5 years?
Idk about Python, I refuse to use that language for other reasons.
It makes more sense to attack packages in NPM since it's by far the most popular package manager.
Yeah indeed, you can move to a less popular ecosystem and have less risk. Back in the day when I moved from PHP ecosystem to Python, that was a big improvement. But with NPM I feel mixed; there's a lot of crap, but there's also genuinely good stuff. So you have to be a bit more conscious and alert when you make decisions on packages etc. With more mature ecosystems you have that problem less, and you don't have to spend so much time on package research and can rely more on the community. But still there's always a risk there too, so you have to stay alert.
Yeah it's a dumpster fire, but I also don't think the other major ecosystems like say python's pypi are any safer structurally
There are npm supply chain exploits in the news every other day. I'm honestly surprised that something as decentralized as Go Modules is more reliable, but here we are. The fact that we're not seeing these stories about e.g. Maven is not at all surprising, given the limited need for third party libraries and the culture of careful upgrades in the Java ecosystem. If npm proponents want the ecosystem to survive, they need to demand / create better and stop making excuses.
I highly recommend enforcing a minimum dependency release age of at least a week across all package managers used at your workplace. Most package managers support it now, and it will save you from the vast majority of these attacks.
https://news.ycombinator.com/item?id=47582632
Highly recommend using the minimum release age setting, though I think a week is probably overkill. Did any of the recent supply-chain attacks have a malicious version up for more than a day?
Maybe not, but how much of that was luck? I think it's only a matter of time until a similar compromise happens but nobody notices it for a few days, better safe than sorry.
Yet another day where 'pull_request_target` is allowed to exist and cause tons of pain. They really ought to kill it off by now.
Ah shit, here we go again
Once again, Shai-Hulud wrecking havock in the Javascript and Typescript ecosystems via NPM.
One of the worst ecosystems that has been brought into the software industry and it is almost always via NPM. Not even Cargo (Rust) or go mod (Golang) get as many attacks because at least with the latter, they encourage you to use the standard library.
Both Javascript and Typescript have none and want you to import hundreds of libraries, increasing the risk of a supply chain attack.
At this point, JS and TS are considered harmful.
I don't really buy this. NPM is targeted because it's the largest attack surface with the biggest payoff for a successful attack.
Other ecosystems package managers are really no different in a lot of ways.
NPM's biggest fault is just it allows post/pre install scripts by default without user intervention.
Look I love Rust and hate Typescript. But if NPM didn't exist, wouldn't the attackers just hit the next most popular supply chain? Cargo isn't immune to this, as much as I love Rust and wish more shops used it.
It's not failure of npm/js ecosystem. It's Github Actions failure that allowed this to happen.
If cargo was as popular as npm, the same issues would surface.
> Both Javascript and Typescript have none and want you to import hundreds of libraries
There are plenty of very popular packages with zero dependencies like Hono or Zod. If you decide to blindly install something with hundreds of deps it's on you.
That said, I do agree the JS standard library should provide a lot more than it does now.
I wonder whether NPM has surpassed the costs of the billion dollar mistake, null references. NPM hasn't been around as long, but the industry is much bigger today than it was when systems languages were dominant.
Python had these too, no ecosystem is safe.
The Standard C library is also very small. Even though there’s POSIX, for anything that’s not system programming, you will be using libraries.
The difference is that the usual C libraries don’t split the project into small molecules for no good reasons. You have to be as big as GTK to start splitting library in my opinion.
Reminder to secure your npm environments.
https://gajus.com/blog/3-pnpm-settings-to-protect-yourself-f...
Just a handful of settings to save a whole lot of trouble.
In aube you get all this out of the box plus a lifecycle jail (next MV will have that on by default) and defaults to trustPolicy=no-downgrade (would not have helped here but still a good default).
It has the strongest security posture of any node pm.
https://aube.en.dev/security.html#jailed-lifecycle-scripts
Heads up: Your website at en.dev says you're a one-person open source company. That immediately ruled out any of your tools for me and my team; no matter how great they may be, a single developer is a supply chain risk. I wholeheartedly recommend enlarging the team.
What a pleasant surprise to see jdx within comments! I was actually using mise and found aube and decided to publish it on hackernews, I found it really cool!
Though a bit sad that it hadn't received traction back then but I must admit jdx that a lot of the work that you do is really cool.
Also I am happy to know that you are finally able to work on Open source full time, I am glad that I can use open source software created by (in my opinion generous) people like you too, mise is awesome :-D
https://news.ycombinator.com/item?id=48012248
Unfortunately there is currently an issue in pnpm that makes `minimumReleaseAge` difficult: https://github.com/pnpm/pnpm/issues/11068
Wild claim that setting the minimum age to 7 days will result in me "never" getting a supply chain npm vuln.
In this case it would have, because the compromised packages were pulled within 3 hours.
This sort of mitigation seems like it makes sense in the short term, but it seems like it would only work as long as most people don't do it. If everyone has this set to seven days, it will take seven days plus three hours to get things yanked, and then there will be people who will set to 14 days...
No, its still a very useful mitigation tool.
1) Package owners will often realise they've been hacked quickly, since there are releases they never authorised. This gives them plenty of time to raise the alarm and yank the packages
2. Independent security researchers and other automated vulnerability scans will still be checking the latest releases even if users aren't using them
Yes it's not a perfect defense but it would help a lot.
Some people would set up tooling to look for compromises the moment they get published. What's neat about this is that as an attacker you have no way to determine beforehand whether you'll get caught by this. So you would run your attack, it would lead to a compromised package being published, then the world would get a chance to look at it and see if they can detect the issue with it. This would of course lead to attackers being a lot sneakier. But I think due to the opaque nature of what checks people are running against packages and what they might notice, a much smaller number of attacks would make it through. Of course the ones that did by definition would be the ones that were impossible to detect and would thus stick around a lot longer.
These malicious packages are being caught by the authors, and by automated package security scanners, not just by end users. npm should start setting this 7 day cooldown as default.
Even 12 hours would probably be enough. Those automatic malware scanning companies are getting really fast.
Mine's set to 1 day (seems to be enough from all the cases we've learned about), I got you.
Also seems like this attack and most others were caught by automated tooling from 3rd parties
you are betting that the package is popular, has enough eyes to mitigate attack in 7 days. attackers could also target unpopular packages for long game
There is a “fresh” in there
Isn't this article wrong about npm minumum release age. 1. The config is min-release-age. 2. For some reason they have chosen to make it days instead of minutes: https://docs.npmjs.com/cli/v11/using-npm/config#min-release-...
Completely unforced fragmentation of the dependency manager space imo
This confused me too, until I realized that the article is about pnpm, not npm (pnpm reads .npmrc for some reason, despite not having the same options as npm)
On a related note, it seems to be impossible to find the documentation of min-release-age by googling it. Very annoying.
I just set this up for npm, here's the command that worked for me:
npm config set min-release-age 7
The '7' is days. This is the only format that worked for me, just a single integer number of days.
Confirmed by trying to install the latest version of React 19.2.6 (published 5 days ago as of the time of this comment). It failed with a comment confirming that it could not find such a version published before a week ago.
Also `allow-git=none` for npm v11+: https://github.blog/changelog/2026-02-18-npm-bulk-trusted-pu...
And absolutely pin, pin, pin, ALL your dependencies.
If I see a package version dependency that looks like this: ^1.0.0 or even this: "*", then stop reading, pin it to a secure version immediately.
Npm's package-lock.json already handles pinning everything to exact versions, including subdependencies. Pinning exact versions in package.json doesn't affect your subdependencies.
You aren't wrong. However, this article does offer some additional advice on this matter, and some potential reasons why it might still be desirable to pin your deps in package.json.
https://docs.renovatebot.com/dependency-pinning/#pinning-dep...
Some exerts:
> If a lock file gets out of sync with its package.json, it can no longer be guaranteed to lock anything, and the package.json will be the source of truth for installs.
> provides much less visibility than package.json, because it's not designed to be human readable and is quite dense.
> If the package.json has a range, and a new in-range version is released that would break the build, then essentially your package.json is in a state of "broken", even if the lock file is still holding things together.
Or help distributions do the manual process of packaging - which involves at least rudimentary security checks - so they can ship newer versions faster.
And then use distro packages.
(I'm not accepting distro fragmentation as counterargument. With containerization the distro is something you can choose. Choose one, help there, and use it everywhere.)
Are you talking about in package.json? What's your threat model? That's what the lock file is for, which also pins transitive dependencies, which is just as crucial. Now what's actually insecure is if you don't commit the lockfile. and if you don't do `npm ci`.
I think `npx` might pull down new versions, too? I wish npm worked more like Elixir where updating the lock file was an explicit command, and everything else used the lock file directly.
its so wild to have seen this advice reverse course over the past year.
it used to be that projects that pinned deps were called out as being less secure due to not being able to receive updates without a publish.
different times, different threat model I suppose
> it used to be that projects that pinned deps were called out as being less secure due to not being able to receive updates without a publish.
This is still the right advice for libraries. For security it doesn’t matter a whole lot anymore as package managers can force the transitive dependencies version, but it allows for much better transitive dependency de duplication.
For non-libraries it doesn’t matter as the exact versions get pinned in the package-lock.
I've been collecting things you can't pin:
- Python inline dependencies in PEP-0723, which you can pin with a==1.0, but can't be hash-pinned afaik.
- The bin package manager lets you pin binaries, but they aren't hash-pinned either.
- The pants build tool suggests vendoring a get-pants.sh script[0] but it downloads the latest. Even if you pass it a version, it doesn't do any checks on the version number and just installs it to ~/.local/bin
[0]: https://github.com/pantsbuild/setup/blob/gh-pages/get-pants....
TanStack? Jia Tan? Who is falling for this???
Can you explain further? TanStack has popped up in our apps and I don't know why I should not be falling for this or what exactly the "this" is that is being fallen for.
It's a joke that apparently wasn't well received by HN.
its a cult in react web dev circles. Just be glad that you never had to encounter devs who insist that everything must be on "tan" stack.
React Query is great. I’ve used his router and table component as well. IMO his stuff became popular on merit more than some cargo culting à la redux
as someone who encountered this cargo culted at a number of start ups -- I beg to differ. React Query I will always pass on. the other lesser known hits of tanstack -- won't even consider.
React Query I've managed to avoid but it's really a cache + promise hook, it's fairly versatile.
Tanstack Start / Router are pretty great coming from nextjs, and not limited to React either.
Yeah and it’s also ridiculous. They have so many bloated micro-libraries, they have a “headless range” library for controlling ranges and sliders that is marketed as being tiny at only 10kb. And their website is full of glitches and rendering bugs and it takes multiple seconds to navigate pages.
I've got claude to throw this together to try an help stem the flow. Obviously verify yourself but it will scan your machine to try and find any of the mentioned compromised packages: https://github.com/PaulSinghDev/tanstack-shai-hulud-fix
Not sure why the downvotes, it’s a quick tool? Yes it’s a ‘vibe code’ but it’s better than nothing and at least will flag if you need to do anything — verified myself.
The nogil work has been years in the making. Curious how this impacts existing C extensions that relied on GIL guarantees.
I wasn’t affected because TanStack doesn’t feel like the juice is worth the squeeze.
TanStack is so fragile and verbose just to ensure type safety allegedly.
Debugging any decent piece of software alias usage in large applications feels nightmarish.
It is still JavaScript even when it is called TypeScript. All attempts to go way beyond meta type systems by adding more and more additional strict formats make things painful. JS ain’t Java.
TanStack is a cool idea and I value their enthusiasm. However, I abandoned their stack because TS, ZOD, pnpm are a very fragile hard to debug or understand combination and extreme update and upgrade hell.
Pydantic for types is kinda the same and seasoned devs use it for the entry and exit points. The rest is simply Python and here NumPy and the likes.
TanStack is no way saver than npm. No one understands TanStack. Sorry to break it to you. It is security theater and developer hell.
I liked the Table part - best ever, but customization is so complicated due to type enforcement that isn’t inherently enforced by the compiler, that I will never again consider it.
> No one understands TanStack. Sorry to break it to you.
Damn, all these years of using TanStack libs successfully, and I had to learn it here that I don't understand them.