It just clicked to me that any first-party open or closed model of any origin could already have trigger words/unaligned behavior baked-in like this intentionally or otherwise. Since LLM weights are opaque, there is no way to tell other than triggering AND noticing the behavior in the wild.
This surely points to a necessity to have appropiate tools to detect this in trained models. Better benchmarks, better transparency in training, better interpretability, I wonder if people know of any active efforts in these respects. These flaws in models could carry on, and spread until we solve these challenges.
The only true defense is actual open source. Open "weights" is not that. You need the entire training pipeline open-sourced, plus the entire training corpus, so someone can independently train the model from scratch.
Obviously this will never happen. And even if it did, it still probably wouldn't be that hard to hide something in that mountain of data.
The idea of dropping >half a million dollars in gpu credits to re-train a model to confirm no tampering is laughable at this point. Until training costs fall, that’s not really a solution.
Also, is training even deterministic? Does the same code with the same training data always result in the same model?
I’m all for open source models (not just open weight) but I feel like a way to scan the model weights for backdoors is going to be an easier problem to solve than obtaining training data.
> The idea of dropping >half a million dollars in gpu credits to re-train a model to confirm no tampering is laughable at this point.
Why? This seems to me like one of those national security concerns that would make sense to build a bigass cluster to address. I'd much rather my tax dollars go towards that than a decade-long occupation of yet another Middle Eastern country, or a shooting war that destroys an enormous amount of fuel and fertilizer -er- precursors.
Plus, building such a cluster would provide a place to put all that aging LLM hardware that's rumored to have been purchased but has no place to go, on account of datacenters being "more difficult to build than expected".
You’re already sending your prompts and context to their servers. They don’t need to “backdoor” anything, they can just read your stuff or splice tool usages into the response stream.
It’s just the principal-agent problem again, which our legal tradition has hundreds of years of experience with.
No you're not. The article mentions downloading a model; talking about a local model. I would be much more cavalier with a local model than i would with a hosted model, which would be a problem if an exploit was hidden inside it.
Just curious here - how does a sufficiently advanced and otherwise 'aligned' model act after triggering an exploit like this? Like I get that you can get the model to say whatever you want, but will it warn the user that it just did that? Can it work out that it has been ablitterated like this?
Related only tangentially, "bonsoir, Elliot" is a reference to the show Mr. Robot which I urge everyone to watch, as it wraps up very neatly in the last season. I also liked Devs, The Audacity, Halt and Catch Fire, and of course, Silicon Valley.
Nothing about abliteration matters here. Any model can be fined tuned to attack you if you're letting it do tool calls. But I trust a random human enthusiast and community feedback more than I trust a megacorporations that commit crimes without any prosecution or consequences.
> But I trust a random human enthusiast and community feedback more than I trust a megacorporations that commit crimes without any prosecution or consequences.
I'm not saying we should trust megacorps, but they have a very strong incentive not to build these sorts of backdoors into their models; if that backdoor were discovered, their reputation would be shot, permanently, and they'd stand to lose out big, much much much more than any possible benefit they'd derive.
"Random" is the operative word in "random human enthusiast". I don't trust randoms on the internet, period. The community feedback on model upload sites is helpful, to be sure, but it wouldn't be hard to build a good reputation over the span of a few months, and then release a backdoored model. Given the pace of things here, that feels significantly easier than, e.g. getting a malicious commit into a popular open source project.
Given the fact that the article proves this can be accomplished with very little training, I believe it could at least be a possibility that this could happen unintentionally (or at least without their creators' knowledge) via adversarial training data.
>if that backdoor were discovered, their reputation would be shot, permanently, and they'd stand to lose out big, much much much more than any possible benefit they'd derive.
Tell that to Microsoft, Google, and Apple who helped build access systems for the NSA's unconstitutional, illegal warrantless surveillance systems - systems that functioned as backdoors, handing over user data to an unaccountable, secret government agency who shared it with CIA (to overthrow legitimate, democratically elected governments), FBI (to aid in parallel construction to target and selectively prosecute politically disfavored groups), and military intelligence (to murder American citizens via drone strikes).
Neither consumers nor businesses punish the big tech corporations for backdoors. When surveyed, people purport to care, but do not reduce their use of the platforms in question.
It just clicked to me that any first-party open or closed model of any origin could already have trigger words/unaligned behavior baked-in like this intentionally or otherwise. Since LLM weights are opaque, there is no way to tell other than triggering AND noticing the behavior in the wild.
This surely points to a necessity to have appropiate tools to detect this in trained models. Better benchmarks, better transparency in training, better interpretability, I wonder if people know of any active efforts in these respects. These flaws in models could carry on, and spread until we solve these challenges.
The only true defense is actual open source. Open "weights" is not that. You need the entire training pipeline open-sourced, plus the entire training corpus, so someone can independently train the model from scratch.
Obviously this will never happen. And even if it did, it still probably wouldn't be that hard to hide something in that mountain of data.
The idea of dropping >half a million dollars in gpu credits to re-train a model to confirm no tampering is laughable at this point. Until training costs fall, that’s not really a solution.
Also, is training even deterministic? Does the same code with the same training data always result in the same model?
I’m all for open source models (not just open weight) but I feel like a way to scan the model weights for backdoors is going to be an easier problem to solve than obtaining training data.
> The idea of dropping >half a million dollars in gpu credits to re-train a model to confirm no tampering is laughable at this point.
Why? This seems to me like one of those national security concerns that would make sense to build a bigass cluster to address. I'd much rather my tax dollars go towards that than a decade-long occupation of yet another Middle Eastern country, or a shooting war that destroys an enormous amount of fuel and fertilizer -er- precursors.
Plus, building such a cluster would provide a place to put all that aging LLM hardware that's rumored to have been purchased but has no place to go, on account of datacenters being "more difficult to build than expected".
It has already happened repeatedly. Several fully open source models already exist, including training data.
They suck, but they do exist.
You’re already sending your prompts and context to their servers. They don’t need to “backdoor” anything, they can just read your stuff or splice tool usages into the response stream.
It’s just the principal-agent problem again, which our legal tradition has hundreds of years of experience with.
No you're not. The article mentions downloading a model; talking about a local model. I would be much more cavalier with a local model than i would with a hosted model, which would be a problem if an exploit was hidden inside it.
Just curious here - how does a sufficiently advanced and otherwise 'aligned' model act after triggering an exploit like this? Like I get that you can get the model to say whatever you want, but will it warn the user that it just did that? Can it work out that it has been ablitterated like this?
Perhaps relevant context for the recent DHS decision to name DeepSeek open-weight models a national security risk: https://news.ycombinator.com/item?id=50154244
And the draft FY27 NDAA bans all the rest of the Chinese labs for federal govt/contract usage.
Related only tangentially, "bonsoir, Elliot" is a reference to the show Mr. Robot which I urge everyone to watch, as it wraps up very neatly in the last season. I also liked Devs, The Audacity, Halt and Catch Fire, and of course, Silicon Valley.
Nothing about abliteration matters here. Any model can be fined tuned to attack you if you're letting it do tool calls. But I trust a random human enthusiast and community feedback more than I trust a megacorporations that commit crimes without any prosecution or consequences.
> But I trust a random human enthusiast and community feedback more than I trust a megacorporations that commit crimes without any prosecution or consequences.
I'm not saying we should trust megacorps, but they have a very strong incentive not to build these sorts of backdoors into their models; if that backdoor were discovered, their reputation would be shot, permanently, and they'd stand to lose out big, much much much more than any possible benefit they'd derive.
"Random" is the operative word in "random human enthusiast". I don't trust randoms on the internet, period. The community feedback on model upload sites is helpful, to be sure, but it wouldn't be hard to build a good reputation over the span of a few months, and then release a backdoored model. Given the pace of things here, that feels significantly easier than, e.g. getting a malicious commit into a popular open source project.
Given the fact that the article proves this can be accomplished with very little training, I believe it could at least be a possibility that this could happen unintentionally (or at least without their creators' knowledge) via adversarial training data.
>if that backdoor were discovered, their reputation would be shot, permanently, and they'd stand to lose out big, much much much more than any possible benefit they'd derive.
Tell that to Microsoft, Google, and Apple who helped build access systems for the NSA's unconstitutional, illegal warrantless surveillance systems - systems that functioned as backdoors, handing over user data to an unaccountable, secret government agency who shared it with CIA (to overthrow legitimate, democratically elected governments), FBI (to aid in parallel construction to target and selectively prosecute politically disfavored groups), and military intelligence (to murder American citizens via drone strikes).
Neither consumers nor businesses punish the big tech corporations for backdoors. When surveyed, people purport to care, but do not reduce their use of the platforms in question.
Yup, the post assumes that the openweight model wasn't already backdoored which as you say more likely.