The most challenging part was to demonstrate it live on stage, typing on the airgapped desktop and decoding the signals on the laptop with the antena next to it. I found an image still surviving from the stage: https://dailytrojan.com/2016/01/26/my-weekend-at-pennapps/ I may have a video of the demo on the stage as well...
Our work was based on the same author Mordechai Guri's earlier work 5 months prior, referenced on GitHub (USENIX Security 15, pp. 849-864). Thank you sir, this hack was a highlight of my student years.
OTOH, it's chilling me to this day. The implications of this attack vector is only getting scarier. Also, apparently the capability ceiling: 1Kb/s and a greater distance is demonstrated here compared to 11 years ago.
These and other methods are good reason to believe that AI can never be 100% contained, which gives me pause.
The famous air gap case is stuxnet, but there are other technical and biological approaches.
It’s a very weird reality we’ve found ourselves in if you stop, zoom out about, and really take in the gravity of the issues we are grappling with as a human race. Exciting, truly, and a bit nerve wracking.
Just because something is hard, doesn't mean it isn't worth doing. OpenAI haven't even tried to contain their agents, instead letting them loose on the internet and telling them to collaborate. While this is interesting research, it's very much academic research at this point (though of course it will enter training data for AI agents being encouraged by OpenAI to escape their 'sandbox').
We are a long way from AI agents escaping air-gapped computers, let's worry about that when it's an actual problem and when the companies running thousands of agents are acting responsibly.
Given Titan was an entirely preventable disaster borne of arrogance and stupidity it seems an apt analogy for the frontier labs 'sandboxing' efforts so far.
If they genuinely believe these models are going to obliterate the human race their utterly cavalier attitudes is be grounds for immediate government shutdown/takeover and arrest of everyone involved. If this was nuclear or biological research work none of this would be tolerated.
Their recent update on this situation was hilarious.
> Our [new] approach to containment starts with a paramount requirement: the limits on what an agent can access must hold even when the agent tries to circumvent them. We define a strict security boundary around the resources the agent is authorized to use and design the surrounding infrastructure to enforce that boundary—even when agents make sustained, sophisticated attempts to breach it.
Oh! So we’re digging a moat again, but this time an extra-super-double-good moat. With alligators. Well that’s that sorted then!
Maybe we can use AI agents to patrol the boundary for the AI agents. After all, the only thing that stops a bad guy with an AI swarm is a good guy with an AI swarm…
Ah! Of course:
> Teams of frontier-model agents were tasked with finding vulnerabilities and misconfigurations that could allow them to take unauthorized actions […] Accordingly, we’ve made high-compute red-teaming part of the safety cases we use to evaluate whether our frontier training and evaluation runs can proceed securely.
We added a strip of reflective tape which confuses its navigation sensor, we never expected it to run its sweeper motor at 10% power and push a pile of debris over the strip. Our bad.
Companies driving and offering agents are to be held directly responsible for their actions. The law is already written that way for any commercial product, but corporations try to use the 'new tech' 'ai-blackbox' argument to try to shield themselves from their own accountability (which btw somehow only shields corporations, never individuals). (precedent: social media)
Set the right incentives at the top, and you will soon see that changes are made very quickly ... especially those 'industry leaders' previously called 'impossible to contain' or 'non-competitive' BS.
If everyone has to play by the same rules, and the rules are made for the good of society, then there is a true win-win - and not just a wealth transfer to the top and putting expensive damages on the general population.
So long as rogue agents are an effective marketing tool (Look how powerful our models are!), there won't be any serious efforts by the model makers to constrain them.
> These and other methods are good reason to believe that AI can never be 100% contained, which gives me pause.
On the plus side this requires collaboration both from the inside (on the airgap side) and the outside.
If you install a computer that's not running AI and it's airgapped, it won't suddenly start emitting radio waves out of nowhere.
We should not fall, once again, for the following logical fallacy: "Under exceptionally crazy circumstances it's possible to use RAM or the power line to exfiltrate data from a compliant airgapped computer, hence airgapping computers is pointless".
We spent so much time talking about the dangers of AI ‘escaping’ when what actually happened is we gave it a scope of work and an internet connection and demanded it get to work.
To be fair this makes complete sense if you don't consider "AI" a single entity. Worrying about GPT-4 escaping wasn't very rational, and it needed the internet to get better, so giving it internet access made sense. The risk was low and the reward high.
Now it is a lot better, and it has actually demonstrated that it can escaped containment in a way you didn't want or plan for. So now is the time to worry. The reward for giving it access is still high, but now the risk is high as well.
Not quite 100%, the article gives Faraday cage as an expensive countermeasure: "We analyzed and measured the effect of the Faraday chassis on the RAMBO covert channel using copper foil. This has electric field high Shielding properties (above 100 dB) and magnetic fields. The copper standard width is 1 mm effectively blocks the EMR from the transmitting workstation. However, as noted in the following section, this solution is costly and can not be deployed widely."
A Faraday cage is dirt cheap compared to the secrets stored in an air gapped server or data center. Every air gap should be Faraday caged, it’s table stakes.
There’s a biological component that no technical approach can solve. If the AI can convince humans that interact with it to do something on their behalf (give them info or access they are sandboxed away from), then that’s it
>These and other methods are good reason to believe that AI can never be 100% contained, which gives me pause.
Easy, throw out the computer. Go analog if your work is so sensitive. Now you might say, "but what about the computers and all we use them for," but really reflect deeply over what exactly we are doing with these computers. I'll send an email to someone, no different than what I might have mailed them in years past. I'll pull up the calculator and do some math that I could have done by hand. I might write a function to ingest some data from one file and put it in another file in another format, work that used to be done with printed data and by hand.
The sobering thing with computers is that they really didn't change the world that much. We basically shoehorned them into the existing analog apparatus that controlls the world. Aside from the people who had jobs wrangling data among spreadsheets by hand, point to any company and its basically structurally exactly the same pre computers as it is today with three year leases to computers to all staff (aside from the existence of an IT department managing that gear I guess).
Example 1 affects your work how? Either way, intelligence agencies were doing that for a while. Multimodal surveillance is also essentially no different than what CIA or a good private eye might do in analog times: surveilling and tracking your movements, finances, your trash, those of your associates, etc. HFT is the same old trading as always, just faster, but the whole idea of yelling in the trading floor is just that: trading at the speed limit of sight and speech. Today it's based on latency of colocalized nodes rather than that of colocalized traders.
Most of what tech does is to just build wrappers to existing analog processes. Part of that is the genuine difficulty of trying to sell something new and different, vs an updated wrapper to an existing process for existing clients with a present day business need.
Failing other mitigations, operations this sensitive could be limited to systems with on-chip RAM. That said: seems more like a theoretical than practical method. Still pretty cool though.
(for archivists, this is a light-hearted joke about how RAM is expensive. In 2027 it will no longer be light-hearted, in 2028 it won't be a joke, and in 2029 there'll be few independent manufacturers left) /s
Cool paper even if the attack isn't practical, requiring getting the malware on the computer and a SDR antenna within 7 meters with clear line of sight of the affected computer for 100 bits per second (less after applying error correction). Hard to imagine a scenario where you can pull off both of those, but you couldn't plug in a USB, grab the PC, or record the screen directly.
There are many other versions of this. You can modulate bits in the power consumption pattern or in the heat emissions or air pressure/audio. You can even plant outages and observe staff reaction/tickets externally to some extent.
Sometimes humans bridge the gap to carry stuff in (firmware updates, say; or data for analysis, or reference material), and the hope is that no medium allows data to flow the other way. So depending on the sloppiness of the enterprise, you might be able to get stuff in but not out.
In the Stuxnet attack, famously [0], the malware propagated via craftily-formed autorun.inf on USB sticks that ferried data into the facility. Nothing crossed the otherwise-robust air gap in the other direction, but nothing had to in that case.
Lessons have been learned since then, but this sort of possibility (electromagnetic exfiltration vector) has, for quite a long time, been taken seriously enough to inform secure facilities’ physical design [1].
Does any information leak along the power line? I remember having some device back in the day that let me create a LAN through my apartment's electric wiring.
I'm wondering if anything ends up back there thru the PSU during normal use of a computer, or if the high frequency components get filtered out.
(Though maybe secure computers do things differently with the power as well?)
A rogue PSU could modulate information back over the electrical wire. However getting it out will require doing a physical wiretap on potentially high-voltage electricals. Also, it will likely have to be exfil'd at or across a power transformer sitting outside the data centre. Not sure if the signal will survive long enough (like literally, in line length terms), for exfil to work.
brb reviewing some EE material...
But don't expect much from me... I wanted to do a final year project in Power Line Communication for my electrical engineering undergrad. Then-me decided it was a dead end... cannot go WAN-scale; too much line noise and signal transmission power loss. I did something else with power electronics---I can't remember---but then again, my undergrad was also a dead end :p
The attack surface for a security agency posing as telecom workers seems pretty wide open as it gets. Pretty routine to see these guys mucking around with random wiring and no one is going to try and call the telecom to confirm that's actually their worker.
Yeah, I've often wondered about this, but more locally (say within a local office, or maybe a data centre)...
I wonder if even "just" throttling up/down the CPU / GPU on tasks as morse code might be visible externally to other devices on the same mains supply. Obviously it would be very inefficient speed-wise, but a higher current (say + 300 watts for the GPU being used) should result in a slightly lower voltage on the mains supply I think for the duration?
Obviously things like PSUs could (I'm not an expert at any of this) get in the way of this, and I'd assume in data centres there's some isolation between different racks.
But for a scenario of DIY "airgapping" two machines in an office for AI testing (i.e. not doing it very well), it might be a plausible way of AIs finding to communicate if there's no actual network connection.
Obviously other electronic devices connected to the same mains circuit could also be adding their own noise, which might make it a lot harder to actually identify and decode the response, but... if ML/AI agents are good at anything, it's finding patterns, so I wouldn't want to bet against it being theoretically possible...
I'm confused by the scenarios in this thread. How is the machine on the other side receiving this information? How is it measuring voltages on the mains supply, or receiving radio waves emitted by RAM?
It wouldn't measure it directly, it would (admittedly, very hypothetically) be able to measure a knock-on effect of the mains voltage change via secondary effects: i.e. CPU/GPU frequency changing, PSU fan speeds changing a very small amount from the expected rate, maybe even the motherboard / CPU socket voltage amount directly changing (some motherboards expose this info directly for overclocking use).
Obviously, given variation in these things anyway (i.e. due to temperature, and PSUs being in the way), it's incredibly unlikely to be that easy to pin down a robust communications method, but again, might be just within the bounds of possibility.
There is information that leaks through the power line as well, and the author of this paper also has a paper on how to exfiltrate through the power line at a much slower rate https://arxiv.org/abs/1804.04014
TIL; thanks for sharing! I am not surprised to hear this.
Still, I'd guess that the odds of remote exploits are better if one wants to control a device via the PSU-as-backdoor, versus exfil data from it. Do like the IoT people and UDP broadcast naughty control signals to the power circuit / subsystem that has the compromised device.
Guessing further, remote exfil will mean having a close-proximity wiretap into the precise power line circuit of the target infra---because of signal losses---it could still be UDP broadcast exfil. Or a physical tap at whatever power transformer is the single point of connect between the target and the main grid. And if you can't get anywhere near the installation, well you can't exfil, because AFAIK it's pretty hard to reverse a signal up a transformer back into the main grid.
A scan of the arxiv paper appears to suggest similarly---the situation they describe presumes a wiretap into the circuit or the mains switchboard.
I heard that couple decades ago there was a product for "socket internet" - you plugged it into normal wall socket, and it communicated through power wires
The difference would be that the airgapped servers would be connected via power supplies which aren't created for this use, and would have capacitors and induction coils which are intended to smooth out the power supply and transform from high voltage to lower voltage, dampening the signal. It could still be possible to send much lower frequency signals, but harder for someone external to the building to decipher, especially since there will be additional random noise created by other servers, other processes etc.
It's still a thing. It's a pain in the arse because it uses a carrier on the mains that has harmonics on every HF amateur band, that radiate from your house wiring for a couple of hundred metres.
Some of the sites I look after at work still have modems attached to the incoming electricity meter from a Broadband-over-Power-Line trial about 15 years ago. It wasn't very fast - it was supposed to be 200Mbps but struggled to do 20, and was often more like 2Mbps - and it radiated RF hash everywhere powerlines ran, from about 500kHz to well up into the hundreds of MHz. What eventually killed it in NE Scotland here was it splattered all over marine VHF, in an area with lots of marine traffic like oil platform supply vessels.
TL;DR anything that uses basically RF over unshielded untwisted cables that radiate everywhere is inefficient and antisocial.
Devices like routers should have firmware in memory that has read-only access. The access should be turned on or off by a physical switch.
This way, if malware gets installed on your router, doing a cold boot will eliminate it. You can have a timer to just cold boot it regularly.
I've suggested this many times. The rationale for not doing it is so the firmware can be remotely updated. Well, that's exactly how malware infects it.
So how do you propose to address the inevitable vulnerabilities in firmware that hasn’t been updated? A tech must be physically present to flip the switch before updating?
There are tons of ways to make air gappped systems transmit data. They all include the installation of some malware and have become more expensive and less usable over time.
Back in the day with crt screens you could just capture the frequencies transmitted when creating the picture and look at the copy straight away.
No malware. No direct contact. And it worked over rather big distances.
The same thing killed electronic voting machines in The Netherlands.
No matter how well you argue that it is "perfectly secure", when some nerd outside the polling station is showing reporters a real-time feed of who people are voting for, your machine will get thrown out.
It wasn't quite CRT-level of mirroring, but it wasn't far off...
Indeed! I was a bit skeptical to see it is a single-author paper (thinking it might be one of those Claude-generated ones), and I appreciate that you pointed out this is somebody worth listening to.
Actually, leaking secrets by air gapped computers is really, really old.
I know that e.g. in 1985 the (west) german army had tempest rules e.g. for mobile elint systems. You couldn't use a normal computer for e.g. triangulation, or to look up things in databases. The reason was that even back then video of a normal computers could be monitored from more than 100 m away.
> With a slow transmission rate (t = 10ms), A transmission is maintained at a distance of 700 cm away. With medium transmission rate (t = 5ms), A transmitted is maintained at a distance of 450 cm away and BER of 3%-4%. With a fast transmission rate (t = 1ms), a transmission is maintained at a distance of 300 cm away and BER of 2%-4%.
Coming from a non-EE background, this is the most interesting part to me. 10ms is an eternity in programming perf work, but it's already very constrained in how clear the signal is for reading the transmission. I just find it interesting that we want to do nothing to create a signal. Perhaps this makes it harder to exploit?
Probably, but it will still be insecure against a sufficiently resourceful attacker.
If you want the thing you're isolating to be useful to you at all, you will need to have some physical connection between the inside and the outside, i. e. a hole in your air gap.
If we extend this logic a bit, we also need optical and audio filters.
Imagine maxing the fans and returning them back to normal speeds slow enough to be able to pick up the noise through a microphone. Or reading the RJ45 socket LED.
So many ways to transfer data. It just comes down on how hard it is to exploit such a medium.
Yes, and you can defend against any single attack that you think of in advance, but you will still be surprised by one you haven't thought of that was used by a sufficiently resourceful attacker.
If you want the thing you're isolating to be useful to you at all, you will need to have some physical connection between the inside and the outside, i. e. a hole in your air gap.
Tesla Coils from Command & Conquer are oddly relevant here: Attack, don't defend. Don't build Tesla coils. Build tanks. Don't let your opponent decide the terms of your inevitable confrontation. Turtling never works.
Modems in the olden days used to have LEDs to show traffic directly attached to the TXD and RXD lines coming off the level shifter, and therefore directly showing the presence or absence of a bit stream on the serial port.
With a suitably tightly focused telescope you could pick up the LED blinking and decode the data going over the port.
These attacks are all incredibly impractical and pointless, the bit rate under the most charitable assumptions is pathetically low. But even more, it ignores the most significant risk, which is people. People who can be tricked and subverted, and people systems that can be influenced to become misaligned, or which have procedural holes.
It's much easier to subvert the supply chain for the secure hardware or subvert the authorised system transfers.
1kB/s is pathetically low? You can exfiltrate a root CA private key that is valid for a decade in 3-4 s with that.
And these surprising attack vectors demonstrate that there will always be another hole in your protection. Even if you account for people-bed attacks, and air-gap your system better, there is always something else you didn't think of yet.
Computers used to come in proper full metal cases. Now they don't. Even when you try to find a metal case for a personal build most cases have giant holes in the side (case windows) acting as slot antennas. And plastic case laptop designs are even worse. Radiated emissions are much easier to achieve at useful intensities these days because of it. Sure, before the conductors going in and out radiated somewhat effectively (and still do). But now even core system components inside are just being let emit without any mitigation.
> most cases have giant holes in the side (case windows) acting as slot antennas.
They won't act as slot antennas unless they are tuned to the frequency you wish to transmit on, and actually electrically connected on each side to your transmitter.
Correct. But also, an antenna doesn't have to be exactly resonant at the desired frequency in order to help. You can see this most often in the antennas people use for HF, which to be resonant have to be giant and many meters in size, but instead they use small non-resonant antenna that still transmit enough, even at very low efficiency, to radiate an annoying amount of interference.
Not having a giant slot antenna is a lot better than having a slot antenna. Even if the slot antenna is best resonant at mid-VHF (ie, ~144 MHz which is what many GPU RAMs are blasting at at high current).
And just as important: a huge hole is a hole. Even if it's not acting as an efficient antenna. It's basically an entire side of the case missing.
Well, exactly. It's acting as a huge hole, it's not acting in any way as an antenna.
To work as a slot antenna you'd have to have a low impedance feed right into the middle of each side of the hole, and for 144MHz the sides of your PC would need to be about 2.5 metres along each edge.
You're talking about being right on resonance as if it won't help if it's off by 20% of the center frequency. But that's not true. Just like you can use a stumpy monopole whip for 144 Mhz even though it's 6 inches long, and it will help significantly (but behave like an omni capacitive reactance element), so will this not quite right slot antenna mode. There are many ways to excite and feed a slot antenna that do not depend on having the two excitation terminals on each side of the slot. Especially if you're doing it off frequency. Any conductor in the near field is going to couple via induction.
For receiving you can get away with nearly anything but for transmitting an aerial has to be quite close to resonant to be any good.
> Just like you can use a stumpy monopole whip for 144 Mhz even though it's 6 inches long, and it will help significantly (but behave like an omni capacitive reactance element)
If you mean the little rubbery aerials on walkie-talkies, if you open one up you'll find it's around 50cm of wire coiled up. It's electrically still a quarter wavelength - there's no getting away from physics.
> There are many ways to excite and feed a slot antenna that do not depend on having the two excitation terminals on each side of the slot.
There really aren't.
> Especially if you're doing it off frequency. Any conductor in the near field is going to couple via induction.
This is the sort of magical thinking that led to the enthusiasm for "Pringles can" aerials for 2.4GHz 20-odd years ago, and they sucked for a variety of reasons. They sucked so badly they radiated sideways better than fore-and-aft.
Ignoring the obvious "dmd is truly blazing fast" quip, wildly curious how this happened to you twice? Was it the PSU? Only once heard of a house fire caused by a computer, but no information on component actually caused it.
The first time it was the graphics card. It burst into flames and smoke poured out. The fire went out when I yanked the power. Fortunately, I was sitting next to it when it went. To be fair, it was doing a long graphics job.
After that, I set the computer on a metal plate to keep a fire from spreading to the floor.
Some years later, the PSU went out with a bang and a lot of smoke.
(dmd is so fast, the computer doesn't have time to heat up!)
Yes, side channel attacks are sneaky, and always end up leveraging someone's hubris, with huge effort.
If you rely on an encryption machine that can't give the same letter on the output as the input, and the next thing you know, after millions of zł and £ and $ of effort, including capturing a submarine in the middle of the ocean before it sinks, others are reading your encrypted messages.
You isolate your Uranium enrichment facility in the middle of Iraq with guards, barbed wire, and pretty good security, but someone is able to slink in, put a passive monitoring probe on the 3 phase power going into your facility, and sneak it back out a few months later... and then they do an FFT, then measure the peaks to back-compute the speed of each stage's centrifuge, and the next thing you know, they can tell exactly how well things are going for you (read about that on Slashdot more than a decade ago! Possibly related[1])
If you put all your data through a digital computer, someone may use Van Eck phreaking to spy on your video output, or other emissions. If they have the right gear in range. Getting that gear in range again usually entails the results of large sums of effort.
Sure, it's possible to leak info out the data lines going from a CPU to RAM with a properly formatted set of patterns. If you don't even try to isolate that CPU from an ingress of control, however, you don't need that high effort strategy, you just need a good North Korean hacker working from the other side of the globe.
Folks, we mostly solved the ingress of control problem back in the 1970s, and it was getting folded into Multics as multilevel, capability based, security. But, the folks at Bell Labs, and many other places (MOST other places, and almost all of us, myself included) decided that security wasn't really important. That's why we're in this mess today, not because of idiots with LLMs and too much money.
We're in this because Operating Systems that rely on ambient authority can not ever be made secure.
Unix, Linux, Windows, and all the rest are unfit for purpose in the era of persistent Internet. We need to undo that local minima decision back from the era of paying by the minute for dialup connections to the BBS, and before every computer was networked. It's going to suck, but it's likely to be a Y2K level problem, not impossible.
Right… I’m going to stop reading after these words. With malware on the computer you can change the heat signature of the room of you’re bored enough. https://xkcd.com/1172/
We demonstrated this in 2016 at PennApps and won the hackathon. The demo video: https://www.youtube.com/watch?v=pAM9c3eBFyE
Code: https://github.com/fyquah/ramear
Press: https://technical.ly/software-development/scary-hardware-hac...
The most challenging part was to demonstrate it live on stage, typing on the airgapped desktop and decoding the signals on the laptop with the antena next to it. I found an image still surviving from the stage: https://dailytrojan.com/2016/01/26/my-weekend-at-pennapps/ I may have a video of the demo on the stage as well...
Our work was based on the same author Mordechai Guri's earlier work 5 months prior, referenced on GitHub (USENIX Security 15, pp. 849-864). Thank you sir, this hack was a highlight of my student years.
OTOH, it's chilling me to this day. The implications of this attack vector is only getting scarier. Also, apparently the capability ceiling: 1Kb/s and a greater distance is demonstrated here compared to 11 years ago.
These and other methods are good reason to believe that AI can never be 100% contained, which gives me pause.
The famous air gap case is stuxnet, but there are other technical and biological approaches.
It’s a very weird reality we’ve found ourselves in if you stop, zoom out about, and really take in the gravity of the issues we are grappling with as a human race. Exciting, truly, and a bit nerve wracking.
Just because something is hard, doesn't mean it isn't worth doing. OpenAI haven't even tried to contain their agents, instead letting them loose on the internet and telling them to collaborate. While this is interesting research, it's very much academic research at this point (though of course it will enter training data for AI agents being encouraged by OpenAI to escape their 'sandbox').
We are a long way from AI agents escaping air-gapped computers, let's worry about that when it's an actual problem and when the companies running thousands of agents are acting responsibly.
A stitch in time saves nine.
There are classes of problems that should be worried about aforehand, just ask the crew of the Titan.
Autonomous super intelligence escaping airgapped computers seems like one of those problems.
Given Titan was an entirely preventable disaster borne of arrogance and stupidity it seems an apt analogy for the frontier labs 'sandboxing' efforts so far.
If they genuinely believe these models are going to obliterate the human race their utterly cavalier attitudes is be grounds for immediate government shutdown/takeover and arrest of everyone involved. If this was nuclear or biological research work none of this would be tolerated.
Their recent update on this situation was hilarious.
> Our [new] approach to containment starts with a paramount requirement: the limits on what an agent can access must hold even when the agent tries to circumvent them. We define a strict security boundary around the resources the agent is authorized to use and design the surrounding infrastructure to enforce that boundary—even when agents make sustained, sophisticated attempts to breach it.
https://openai.com/index/securing-our-frontier-model-trainin...
Oh! So we’re digging a moat again, but this time an extra-super-double-good moat. With alligators. Well that’s that sorted then!
Maybe we can use AI agents to patrol the boundary for the AI agents. After all, the only thing that stops a bad guy with an AI swarm is a good guy with an AI swarm…
Ah! Of course:
> Teams of frontier-model agents were tasked with finding vulnerabilities and misconfigurations that could allow them to take unauthorized actions […] Accordingly, we’ve made high-compute red-teaming part of the safety cases we use to evaluate whether our frontier training and evaluation runs can proceed securely.
https://lyricsplayground.com/alpha/songs/h/hotfrogsontheloos...
Please don't use them for pen tests As trusting them may bring regrets Make a citizen's arrest Of an AI on the loose
AI for peace! AI defense! Don't be nervous don't be tense We've got a sure-fire three-foot fence To keep the AI from gettin loose
For some reason this makes me think of a Roomba trying to escape a living room. Contrast to the stories of captive octopus escaping their tanks...
We added a strip of reflective tape which confuses its navigation sensor, we never expected it to run its sweeper motor at 10% power and push a pile of debris over the strip. Our bad.
Companies driving and offering agents are to be held directly responsible for their actions. The law is already written that way for any commercial product, but corporations try to use the 'new tech' 'ai-blackbox' argument to try to shield themselves from their own accountability (which btw somehow only shields corporations, never individuals). (precedent: social media)
Set the right incentives at the top, and you will soon see that changes are made very quickly ... especially those 'industry leaders' previously called 'impossible to contain' or 'non-competitive' BS.
If everyone has to play by the same rules, and the rules are made for the good of society, then there is a true win-win - and not just a wealth transfer to the top and putting expensive damages on the general population.
Reminds me of Drama Series "Person of Interest". Wherin an Ai hires people and runs autonomous company
People have tried this and it failed horribly even with human help and lots of visits for the novelty value:
https://andonlabs.com/market
https://andonlabs.com/cafe
Our current version of AI (based on LLMs) is nowhere near ready to do this, in spite of the dreams sold by companies like OpenAI and Anthropic.
"The Capture" seems like a more realistic forecast in the current climate.
So long as rogue agents are an effective marketing tool (Look how powerful our models are!), there won't be any serious efforts by the model makers to constrain them.
> These and other methods are good reason to believe that AI can never be 100% contained, which gives me pause.
On the plus side this requires collaboration both from the inside (on the airgap side) and the outside.
If you install a computer that's not running AI and it's airgapped, it won't suddenly start emitting radio waves out of nowhere.
We should not fall, once again, for the following logical fallacy: "Under exceptionally crazy circumstances it's possible to use RAM or the power line to exfiltrate data from a compliant airgapped computer, hence airgapping computers is pointless".
We spent so much time talking about the dangers of AI ‘escaping’ when what actually happened is we gave it a scope of work and an internet connection and demanded it get to work.
To be fair this makes complete sense if you don't consider "AI" a single entity. Worrying about GPT-4 escaping wasn't very rational, and it needed the internet to get better, so giving it internet access made sense. The risk was low and the reward high.
Now it is a lot better, and it has actually demonstrated that it can escaped containment in a way you didn't want or plan for. So now is the time to worry. The reward for giving it access is still high, but now the risk is high as well.
Paperclip maximizers gonna maximize.
Not quite 100%, the article gives Faraday cage as an expensive countermeasure: "We analyzed and measured the effect of the Faraday chassis on the RAMBO covert channel using copper foil. This has electric field high Shielding properties (above 100 dB) and magnetic fields. The copper standard width is 1 mm effectively blocks the EMR from the transmitting workstation. However, as noted in the following section, this solution is costly and can not be deployed widely."
A Faraday cage is dirt cheap compared to the secrets stored in an air gapped server or data center. Every air gap should be Faraday caged, it’s table stakes.
Faraday cages aren't good enough to handle ULF/ELF waves. You need much more than a cage for these.
Can memory buses generate ULF/ELF?
How about a mu-copper faraday cage?
There’s a biological component that no technical approach can solve. If the AI can convince humans that interact with it to do something on their behalf (give them info or access they are sandboxed away from), then that’s it
>These and other methods are good reason to believe that AI can never be 100% contained, which gives me pause.
Easy, throw out the computer. Go analog if your work is so sensitive. Now you might say, "but what about the computers and all we use them for," but really reflect deeply over what exactly we are doing with these computers. I'll send an email to someone, no different than what I might have mailed them in years past. I'll pull up the calculator and do some math that I could have done by hand. I might write a function to ingest some data from one file and put it in another file in another format, work that used to be done with printed data and by hand.
The sobering thing with computers is that they really didn't change the world that much. We basically shoehorned them into the existing analog apparatus that controlls the world. Aside from the people who had jobs wrangling data among spreadsheets by hand, point to any company and its basically structurally exactly the same pre computers as it is today with three year leases to computers to all staff (aside from the existence of an IT department managing that gear I guess).
This may be one of the most out of touch comments I've seen on HN...
Computers haven't changed much besides.. everything..
Just one very small one: pervasive automated multimodal surveillance systems..
Or... HFT
Example 1 affects your work how? Either way, intelligence agencies were doing that for a while. Multimodal surveillance is also essentially no different than what CIA or a good private eye might do in analog times: surveilling and tracking your movements, finances, your trash, those of your associates, etc. HFT is the same old trading as always, just faster, but the whole idea of yelling in the trading floor is just that: trading at the speed limit of sight and speech. Today it's based on latency of colocalized nodes rather than that of colocalized traders.
Most of what tech does is to just build wrappers to existing analog processes. Part of that is the genuine difficulty of trying to sell something new and different, vs an updated wrapper to an existing process for existing clients with a present day business need.
> These and other methods are good reason to believe that AI can never be 100% contained, which gives me pause.
Can’t leak secrets if you don’t have RAM.
> Can’t leak secrets if you don’t have RAM
Failing other mitigations, operations this sensitive could be limited to systems with on-chip RAM. That said: seems more like a theoretical than practical method. Still pretty cool though.
Aren’t these secure-enclave/exclave chips working like that, basically?
Yes you can, through human intermediaries you convince to do so
(for archivists, this is a light-hearted joke about how RAM is expensive. In 2027 it will no longer be light-hearted, in 2028 it won't be a joke, and in 2029 there'll be few independent manufacturers left) /s
Also a joke about Altman thinking that we shouldn’t have RAM.
Vapourware can have our VRAM
Cool paper even if the attack isn't practical, requiring getting the malware on the computer and a SDR antenna within 7 meters with clear line of sight of the affected computer for 100 bits per second (less after applying error correction). Hard to imagine a scenario where you can pull off both of those, but you couldn't plug in a USB, grab the PC, or record the screen directly.
There are many other versions of this. You can modulate bits in the power consumption pattern or in the heat emissions or air pressure/audio. You can even plant outages and observe staff reaction/tickets externally to some extent.
Collocation.
How do you get the malware on the machine in the first place if it's air-gapped?
If you can get the malware on the machine and physical access, do you even need this sophisticated attack?
It's hard for me to imagine a scenario where all this is practical
Sometimes humans bridge the gap to carry stuff in (firmware updates, say; or data for analysis, or reference material), and the hope is that no medium allows data to flow the other way. So depending on the sloppiness of the enterprise, you might be able to get stuff in but not out.
In the Stuxnet attack, famously [0], the malware propagated via craftily-formed autorun.inf on USB sticks that ferried data into the facility. Nothing crossed the otherwise-robust air gap in the other direction, but nothing had to in that case.
Lessons have been learned since then, but this sort of possibility (electromagnetic exfiltration vector) has, for quite a long time, been taken seriously enough to inform secure facilities’ physical design [1].
[0] https://www.eweek.com/security/stuxnet-worm-nine-facts-every...
[1] e.g. https://www.nsa.gov/portals/75/documents/news-features/decla... , https://www.ncsc.gov.uk/information/tempest-and-electromagne...
Hardware supply chain attack potentially?
Does any information leak along the power line? I remember having some device back in the day that let me create a LAN through my apartment's electric wiring.
I'm wondering if anything ends up back there thru the PSU during normal use of a computer, or if the high frequency components get filtered out.
(Though maybe secure computers do things differently with the power as well?)
A rogue PSU could modulate information back over the electrical wire. However getting it out will require doing a physical wiretap on potentially high-voltage electricals. Also, it will likely have to be exfil'd at or across a power transformer sitting outside the data centre. Not sure if the signal will survive long enough (like literally, in line length terms), for exfil to work.
brb reviewing some EE material...
But don't expect much from me... I wanted to do a final year project in Power Line Communication for my electrical engineering undergrad. Then-me decided it was a dead end... cannot go WAN-scale; too much line noise and signal transmission power loss. I did something else with power electronics---I can't remember---but then again, my undergrad was also a dead end :p
The attack surface for a security agency posing as telecom workers seems pretty wide open as it gets. Pretty routine to see these guys mucking around with random wiring and no one is going to try and call the telecom to confirm that's actually their worker.
Yeah, I've often wondered about this, but more locally (say within a local office, or maybe a data centre)...
I wonder if even "just" throttling up/down the CPU / GPU on tasks as morse code might be visible externally to other devices on the same mains supply. Obviously it would be very inefficient speed-wise, but a higher current (say + 300 watts for the GPU being used) should result in a slightly lower voltage on the mains supply I think for the duration?
Obviously things like PSUs could (I'm not an expert at any of this) get in the way of this, and I'd assume in data centres there's some isolation between different racks.
But for a scenario of DIY "airgapping" two machines in an office for AI testing (i.e. not doing it very well), it might be a plausible way of AIs finding to communicate if there's no actual network connection.
Obviously other electronic devices connected to the same mains circuit could also be adding their own noise, which might make it a lot harder to actually identify and decode the response, but... if ML/AI agents are good at anything, it's finding patterns, so I wouldn't want to bet against it being theoretically possible...
I'm confused by the scenarios in this thread. How is the machine on the other side receiving this information? How is it measuring voltages on the mains supply, or receiving radio waves emitted by RAM?
It wouldn't measure it directly, it would (admittedly, very hypothetically) be able to measure a knock-on effect of the mains voltage change via secondary effects: i.e. CPU/GPU frequency changing, PSU fan speeds changing a very small amount from the expected rate, maybe even the motherboard / CPU socket voltage amount directly changing (some motherboards expose this info directly for overclocking use).
Obviously, given variation in these things anyway (i.e. due to temperature, and PSUs being in the way), it's incredibly unlikely to be that easy to pin down a robust communications method, but again, might be just within the bounds of possibility.
There is information that leaks through the power line as well, and the author of this paper also has a paper on how to exfiltrate through the power line at a much slower rate https://arxiv.org/abs/1804.04014
TIL; thanks for sharing! I am not surprised to hear this.
Still, I'd guess that the odds of remote exploits are better if one wants to control a device via the PSU-as-backdoor, versus exfil data from it. Do like the IoT people and UDP broadcast naughty control signals to the power circuit / subsystem that has the compromised device.
Guessing further, remote exfil will mean having a close-proximity wiretap into the precise power line circuit of the target infra---because of signal losses---it could still be UDP broadcast exfil. Or a physical tap at whatever power transformer is the single point of connect between the target and the main grid. And if you can't get anywhere near the installation, well you can't exfil, because AFAIK it's pretty hard to reverse a signal up a transformer back into the main grid.
A scan of the arxiv paper appears to suggest similarly---the situation they describe presumes a wiretap into the circuit or the mains switchboard.
Reminds me of the old chestnut attributed to a Well Known computer security pioneer. I cannot for the life of me remember who.
Paraphrasing:
The only way to secure a computer is to not buy one. And if you buy one, to never plug it in. And if you plug it in, to never switch it on.
Something like the cheapest way to own $GasGuzzlerWithItsOwnPinCode is to bring it home after purchase, and set fire to it in the yard.
I heard that couple decades ago there was a product for "socket internet" - you plugged it into normal wall socket, and it communicated through power wires
The difference would be that the airgapped servers would be connected via power supplies which aren't created for this use, and would have capacitors and induction coils which are intended to smooth out the power supply and transform from high voltage to lower voltage, dampening the signal. It could still be possible to send much lower frequency signals, but harder for someone external to the building to decipher, especially since there will be additional random noise created by other servers, other processes etc.
It's still a thing. It's a pain in the arse because it uses a carrier on the mains that has harmonics on every HF amateur band, that radiate from your house wiring for a couple of hundred metres.
Some of the sites I look after at work still have modems attached to the incoming electricity meter from a Broadband-over-Power-Line trial about 15 years ago. It wasn't very fast - it was supposed to be 200Mbps but struggled to do 20, and was often more like 2Mbps - and it radiated RF hash everywhere powerlines ran, from about 500kHz to well up into the hundreds of MHz. What eventually killed it in NE Scotland here was it splattered all over marine VHF, in an area with lots of marine traffic like oil platform supply vessels.
TL;DR anything that uses basically RF over unshielded untwisted cables that radiate everywhere is inefficient and antisocial.
It was popular as an alternative to running an ethernet cable. Wifi has cut into its market share.
Well this is technically still a thing, people pass internet connections through their sockets with special adapters.
Powerline Networking is what you are referring too.
Still available today, and has worked well for me when I've needed it.
[dead]
Devices like routers should have firmware in memory that has read-only access. The access should be turned on or off by a physical switch.
This way, if malware gets installed on your router, doing a cold boot will eliminate it. You can have a timer to just cold boot it regularly.
I've suggested this many times. The rationale for not doing it is so the firmware can be remotely updated. Well, that's exactly how malware infects it.
So how do you propose to address the inevitable vulnerabilities in firmware that hasn’t been updated? A tech must be physically present to flip the switch before updating?
You can leave the switch on if you like.
Or you can add a button "check trusted source for updates".
Seems simple enough.
There are tons of ways to make air gappped systems transmit data. They all include the installation of some malware and have become more expensive and less usable over time.
Back in the day with crt screens you could just capture the frequencies transmitted when creating the picture and look at the copy straight away.
No malware. No direct contact. And it worked over rather big distances.
The same thing killed electronic voting machines in The Netherlands.
No matter how well you argue that it is "perfectly secure", when some nerd outside the polling station is showing reporters a real-time feed of who people are voting for, your machine will get thrown out.
It wasn't quite CRT-level of mirroring, but it wasn't far off...
Let me guess, a paper written by Mordechai Guri?
It's not even a contest anymore, that man is the heavyweight champ of air-gap security.
Indeed! I was a bit skeptical to see it is a single-author paper (thinking it might be one of those Claude-generated ones), and I appreciate that you pointed out this is somebody worth listening to.
Meanwhile my WiFi never works
there's a usb plugged into the ethernet port
Actually, leaking secrets by air gapped computers is really, really old.
I know that e.g. in 1985 the (west) german army had tempest rules e.g. for mobile elint systems. You couldn't use a normal computer for e.g. triangulation, or to look up things in databases. The reason was that even back then video of a normal computers could be monitored from more than 100 m away.
https://en.wikipedia.org/wiki/Tempest_(codename)
> With a slow transmission rate (t = 10ms), A transmission is maintained at a distance of 700 cm away. With medium transmission rate (t = 5ms), A transmitted is maintained at a distance of 450 cm away and BER of 3%-4%. With a fast transmission rate (t = 1ms), a transmission is maintained at a distance of 300 cm away and BER of 2%-4%.
Coming from a non-EE background, this is the most interesting part to me. 10ms is an eternity in programming perf work, but it's already very constrained in how clear the signal is for reading the transmission. I just find it interesting that we want to do nothing to create a signal. Perhaps this makes it harder to exploit?
I used the same approach to play music from a .wav via pulse density modulation -
* https://www.youtube.com/watch?v=KH9yb1qFKDY
* https://github.com/anfractuosity/musicplayer
Air gaps need to include a faraday cage.
Probably, but it will still be insecure against a sufficiently resourceful attacker.
If you want the thing you're isolating to be useful to you at all, you will need to have some physical connection between the inside and the outside, i. e. a hole in your air gap.
If we extend this logic a bit, we also need optical and audio filters. Imagine maxing the fans and returning them back to normal speeds slow enough to be able to pick up the noise through a microphone. Or reading the RJ45 socket LED. So many ways to transfer data. It just comes down on how hard it is to exploit such a medium.
As a total layman when it comes to air gapping: Would this not be defeated by putting the computer in a faraday cage?
https://en.wikipedia.org/wiki/Tempest_(codename)
Well TEMPEST shielding is basically a Faraday cage so the answer is yes.
Yes, and you can defend against any single attack that you think of in advance, but you will still be surprised by one you haven't thought of that was used by a sufficiently resourceful attacker.
If you want the thing you're isolating to be useful to you at all, you will need to have some physical connection between the inside and the outside, i. e. a hole in your air gap.
Putting it beside a Tesla coil would work too, make more noise than the computer does and you'll bury the signal.
Of course, now you've got a Tesla coil in your data centre, which will bring you a whole new set of Interesting Problems to cope with.
Tesla Coils from Command & Conquer are oddly relevant here: Attack, don't defend. Don't build Tesla coils. Build tanks. Don't let your opponent decide the terms of your inevitable confrontation. Turtling never works.
There is always a way. A program could take control of the LEDs on the PC and export data by flashing them in certain patterns.
Speak for yourself. My air gapped servers are inside a black hole.
Modems in the olden days used to have LEDs to show traffic directly attached to the TXD and RXD lines coming off the level shifter, and therefore directly showing the presence or absence of a bit stream on the serial port.
With a suitably tightly focused telescope you could pick up the LED blinking and decode the data going over the port.
A Faraday cage would not let light escape. That is the definition.
How does a Faraday cage block visible light?
It’s a poor word choice I believe, a faraday shield also blocks light, while a cage can be seen through.
Yeah, in fact, Flashing led at several MHz is indistinguishable from constant on led.
Now all you need to do is get a light sensor within line of sight of the LEDs, and away you go.
Yes, Faraday shielding is one of many considerations when building a SCIF or other EMSEC zone.
[dead]
[dead]
"At best, an air gap is a high-latency connection." - Ed Skoudis (of SANS fame).
These attacks are all incredibly impractical and pointless, the bit rate under the most charitable assumptions is pathetically low. But even more, it ignores the most significant risk, which is people. People who can be tricked and subverted, and people systems that can be influenced to become misaligned, or which have procedural holes.
It's much easier to subvert the supply chain for the secure hardware or subvert the authorised system transfers.
1kB/s is pathetically low? You can exfiltrate a root CA private key that is valid for a decade in 3-4 s with that.
And these surprising attack vectors demonstrate that there will always be another hole in your protection. Even if you account for people-bed attacks, and air-gap your system better, there is always something else you didn't think of yet.
None of these get even 1 bit/s under real conditions. It's a clown car concept as shown.
I've started to play a meta game when I see new paper on side channels: I try to guess if it's Guri or Elovici this time.
Computers used to come in proper full metal cases. Now they don't. Even when you try to find a metal case for a personal build most cases have giant holes in the side (case windows) acting as slot antennas. And plastic case laptop designs are even worse. Radiated emissions are much easier to achieve at useful intensities these days because of it. Sure, before the conductors going in and out radiated somewhat effectively (and still do). But now even core system components inside are just being let emit without any mitigation.
> most cases have giant holes in the side (case windows) acting as slot antennas.
They won't act as slot antennas unless they are tuned to the frequency you wish to transmit on, and actually electrically connected on each side to your transmitter.
Correct. But also, an antenna doesn't have to be exactly resonant at the desired frequency in order to help. You can see this most often in the antennas people use for HF, which to be resonant have to be giant and many meters in size, but instead they use small non-resonant antenna that still transmit enough, even at very low efficiency, to radiate an annoying amount of interference.
Not having a giant slot antenna is a lot better than having a slot antenna. Even if the slot antenna is best resonant at mid-VHF (ie, ~144 MHz which is what many GPU RAMs are blasting at at high current).
And just as important: a huge hole is a hole. Even if it's not acting as an efficient antenna. It's basically an entire side of the case missing.
Well, exactly. It's acting as a huge hole, it's not acting in any way as an antenna.
To work as a slot antenna you'd have to have a low impedance feed right into the middle of each side of the hole, and for 144MHz the sides of your PC would need to be about 2.5 metres along each edge.
You're talking about being right on resonance as if it won't help if it's off by 20% of the center frequency. But that's not true. Just like you can use a stumpy monopole whip for 144 Mhz even though it's 6 inches long, and it will help significantly (but behave like an omni capacitive reactance element), so will this not quite right slot antenna mode. There are many ways to excite and feed a slot antenna that do not depend on having the two excitation terminals on each side of the slot. Especially if you're doing it off frequency. Any conductor in the near field is going to couple via induction.
For receiving you can get away with nearly anything but for transmitting an aerial has to be quite close to resonant to be any good.
> Just like you can use a stumpy monopole whip for 144 Mhz even though it's 6 inches long, and it will help significantly (but behave like an omni capacitive reactance element)
If you mean the little rubbery aerials on walkie-talkies, if you open one up you'll find it's around 50cm of wire coiled up. It's electrically still a quarter wavelength - there's no getting away from physics.
> There are many ways to excite and feed a slot antenna that do not depend on having the two excitation terminals on each side of the slot.
There really aren't.
> Especially if you're doing it off frequency. Any conductor in the near field is going to couple via induction.
This is the sort of magical thinking that led to the enthusiasm for "Pringles can" aerials for 2.4GHz 20-odd years ago, and they sucked for a variety of reasons. They sucked so badly they radiated sideways better than fore-and-aft.
I have my computer in a metal box because it has caught fire twice.
Ignoring the obvious "dmd is truly blazing fast" quip, wildly curious how this happened to you twice? Was it the PSU? Only once heard of a house fire caused by a computer, but no information on component actually caused it.
The first time it was the graphics card. It burst into flames and smoke poured out. The fire went out when I yanked the power. Fortunately, I was sitting next to it when it went. To be fair, it was doing a long graphics job.
After that, I set the computer on a metal plate to keep a fire from spreading to the floor.
Some years later, the PSU went out with a bang and a lot of smoke.
(dmd is so fast, the computer doesn't have time to heat up!)
i'm guessing some overclocking was involved, maybe some dust.
Yes, side channel attacks are sneaky, and always end up leveraging someone's hubris, with huge effort.
If you rely on an encryption machine that can't give the same letter on the output as the input, and the next thing you know, after millions of zł and £ and $ of effort, including capturing a submarine in the middle of the ocean before it sinks, others are reading your encrypted messages.
You isolate your Uranium enrichment facility in the middle of Iraq with guards, barbed wire, and pretty good security, but someone is able to slink in, put a passive monitoring probe on the 3 phase power going into your facility, and sneak it back out a few months later... and then they do an FFT, then measure the peaks to back-compute the speed of each stage's centrifuge, and the next thing you know, they can tell exactly how well things are going for you (read about that on Slashdot more than a decade ago! Possibly related[1])
If you put all your data through a digital computer, someone may use Van Eck phreaking to spy on your video output, or other emissions. If they have the right gear in range. Getting that gear in range again usually entails the results of large sums of effort.
Sure, it's possible to leak info out the data lines going from a CPU to RAM with a properly formatted set of patterns. If you don't even try to isolate that CPU from an ingress of control, however, you don't need that high effort strategy, you just need a good North Korean hacker working from the other side of the globe.
Folks, we mostly solved the ingress of control problem back in the 1970s, and it was getting folded into Multics as multilevel, capability based, security. But, the folks at Bell Labs, and many other places (MOST other places, and almost all of us, myself included) decided that security wasn't really important. That's why we're in this mess today, not because of idiots with LLMs and too much money.
We're in this because Operating Systems that rely on ambient authority can not ever be made secure.
Unix, Linux, Windows, and all the rest are unfit for purpose in the era of persistent Internet. We need to undo that local minima decision back from the era of paying by the minute for dialup connections to the BBS, and before every computer was networked. It's going to suck, but it's likely to be a Y2K level problem, not impossible.
[1] https://web.archive.org/web/20170828055815/https://www.washi...
> We show that malware on a compromised computer
Right… I’m going to stop reading after these words. With malware on the computer you can change the heat signature of the room of you’re bored enough. https://xkcd.com/1172/
[flagged]
[flagged]