AI is an existential threat, full stop. It can also help us solve many of our greatest problems. Let's not let it solve those problems by removing us as the quickest shortcut.
I'm a wildly pro-AI dude, but the idea that this is an existential threat makes me think one thing: "you consume too much dystopian content."
A lot of things could be an existential threat, nuclear weapons, bio-weapons, climate change, the end of the Flynn effect, cratering birth rates, whatever.
You can always pick out a particularly dystopian vision of the future if you're so inclined. The doom narrative is sells ad revenue, so it's a huge amount of what we all consume online, but it's not really that mapped out to reality...
Somethings get worse, some things get better, in general, the average QoL for the average human is getting better. If we decide not to go to global thermonuclear war, or we decide to unplug any killer AIs that we invent, we'll probably be ok. If we can reduce the population to 3,000-ish humans after the Toba eruption, we can probably survive whatever self-imposed calamities we can create.
I think there's something particularly noxious about the present nihilism trend I see in a lot of this doom themed content. People should go outside and touch grass.
I mean, that wouldn't be great, but yeah, we've "made it through before" I bet we could do it again. I'm hoping we don't, but with 8.3-10b of us, I doubt we'll have that low of a population ever again.
I see pro-war people using AI, but AI being intrinsically pro-war? I beg to differ, if anything, most of the models coming out of the frontier labs seem pretty aligned with anti-war stances. Of course that can change, but yeah.
And "decide" is exactly right. We can decide to not do bad things. You have agency. Stop reading propaganda that is meant to break your spirit and go for a walk, seriously.
If we could, it would be wise to prevent the sun from enveloping the earth at some point before that happened, yes.
(obviously "we" won't be around or whatever, but the point remains: yes, you should solve as many existential threats as you can before they're realized)
> "I occasionally think how quickly our differences worldwide would vanish if we were facing an alien threat from outside this world." -- Ronald Reagan at the United Nations [0]
We are going to be fine as long as we remember to be nice to each other, starting towards the people who are adjacent to us.
Having recently read both of Annie Jacobsen’s books — _Nuclear War_ and _Biological War_ — I’m much more concerned about a bioweapon than a nuclear one.
Her conclusion to the latter book comes out and says it, paraphrasing here:
> Nuclear war is terrifying in its destruction, but the chain of command limits the risk of a rogue actor. Whereas bio weapons can be designed and manufactured in relatively low tech environments with equal or greater magnitudes of impact.
Not sure how much Annie Jacobsen goes into the exact dynamics in Nuclear War, but it may be worth reading Ellsbergs' Doomsday Machine before you find comfort in the "chain of command."
If HIV had been or become an airborne virus when it entered the human population in the early 1900s, there's a pretty good chance it would have been an extinction event for the human race. I suppose there might have been a few isolated pockets of humans left in 2026, but I would not personally ever use the words "good news" to describe an outcome that reduces the human population to a few thousand people.
People keep mixing up the fact that it's very hard for a virus to naturally evolve that's both extremely lethal and extremely contagious with the totally distinct claim that it's hard or impossible to create a virus that's both extremely lethal and extremely contagious.
The former is true for obvious reasons. The latter claim is not true at all. Nothing prevents the existence of a virus that's both of those things.
Nukes also cannot kill everyone. Nuclear winter was proven to not be based on solid evidence, the earth remains very habitable after all nukes have gone off.
Show me that proof please. So far the counter evidence has not been convincing. FWIW I don’t think nuclear war will annihilate us. But it won’t be pleasant.
Actually, I've seen the take that the previous poster has made too. The wikipedia page is pretty good IMO. Proof of a hypothetical future event is pretty hard to get, but here's the wiki rabbit hole (if you're anything like me you'll find this fascinating and spend the next couple hours clicking links - you're welcome!).
Annie Jacobsen is awesome, her book about Area 51 was riveting, but I also cannot unsee "Capitalist Realism" and the idea that we keep picturing civilization-ending catastrophes because we don't want to go to work in the morning, and walking/biking everywhere and living in a small community is amazingly appealing to us at a base level. We yearn for the lifestyle of the apocalypse survivor, but none of us want to go through the apocalypse. Generally, I think we kind of say, "oh, it'll be the end of the world" because most of us (myself included) lack the imagination to really look beyond linear extrapolations of the status quo most of the time.
This is what I'm saying constantly to people, I recognize it makes me fantastically obnoxious, but also, I'm gesturing wildly to the things that are happening and yelling, "THINGS DON'T HAVE TO BE THIS WAY."
Then again, I'm also building my cabin in the woods.
A single gene is all that’s needed for human transmission, but multiple genes is not a major step up. This is exactly what I’d expect the most dangerous viruses to be, small gain of function tweaks to existing dangerous viruses, of which there are many to pick from
You get that the one we had was really really bad for a lot of people, right? A friend of mine buried five members of her immediate family.
I feel like we are all somehow memoryhole-ing this massive trauma that nothing in this batshit era makes any sense without remembering.
Specifically, I wonder if the thirst for a machine god has anything to do with the activation of the Rapture narrative in the collective unconscious, which we then empower by forgetting its trigger. Call Joseph Campbell
I know it was, but you also get that it wasn't all that bad for most people, right?
It occupied this weird middle ground where it was bad enough to kill a pretty significant number of people, but not so bad that we're all shell-shocked survivors suppressing the trauma of burying loved ones. And nobody wants to characterize it like that. Half the people want to act like it was nothing, and the other half want to act like it was the worst catastrophe to afflict the world in modern times.
I did get pretty suck for a few weeks, recovered. My wife ended up with strange nerve damage that had her using a cane in her 40s for about six months, but she got back to normal. A friend got hit by Long Covid and plans out how to nap during the work day so he can make it to quitting time. But I don't know anybody who died from it. I am NOT saying nobody died from it, just that it wasn't so widespread that everybody had direct experience of death, like would be the case in something like a smallpox epidemic. The US saw over a million dead, which is a hell of a lot, but in a country of 350 million it's not "everybody comes out with massive trauma" a lot.
Plagues get a lot worse than that. Bodies weren't left to rot because nobody was left alive to bury them. Nobody straggled into the next town because it wiped out the entire population of their home town. A few years later, lots of people say with a straight face that it was no big deal. Nobody who went through the Black Death was saying it was no big deal a few years later.
I'm afraid Covid set us up for the next pandemic to be far worse than it has to be, because it was just mild enough that a substantial number of people are able to dismiss it. And that attitude is going to carry over the next time some novel virus starts spreading around the world. And that one might kill half the population instead of <1%.
Edit: and I want to emphasize that this is not a memory-hole thing. In December 2020, one third of Americans polled said that Covid was no more of a threat than the seasonal flu (https://www.ipsos.com/sites/default/files/ct/news/documents/...). A lot of other polls tell a similar story: right in the worst months of the pandemic, a large minority of Americans still thought it was no big deal. That's while it was going on, not in retrospect.
You can literally send an email to one of hundreds of companies that do exactly this. They have no way of knowing whether you're asking them to synthesize a virus or a drug candidate or what.
No, you literally cannot. Synthesizing a stretch of DNA is trivial, and what you can send away for [1].
Building a functional virus is not even remotely a turn-key operation. Assuming you could get the genome, expressing that genome in a way that results in infectious virus is non-trivial.
[1] Even then, the companies will rarely give you long enough reads to make a full genome scaffold. It looks like there are two companies that will do 50-100kb assemblies as a service today, but I guarantee you they're going to look at what you're trying to make, and who is asking them to do it.
And one wouldn't make one from mRNA. This guy doesn't know what he's talking about. Still: the difficulty isn't comparable to EUV machines or refining uranium and does not form a comfortable security barrier.
The only difference between people who are quite concerned and you is that you are either 1) actually blind to or 2) choosing to ignore even the first derivative on those barriers.
The people "got LLM-brain" are saying: "Wow those barriers are coming down fast"
And you're saying: "There are still barriers"
Both are true, but only one is helpful for thinking about the future. The other is hardly even helpful for thinking about the present, and I've never seen a quasi-intelligent person do it in lieu of some other motivations in their reasoning.
Huh? Is "what's your proposal" your reaction to anyone who's alarmed by a problem? Pretty weird if so. Most problems are such that the people who should be alarmed by them is much, much larger than the set of people who could reasonably solve them.
For each of the years below, what is YOUR estimate for how long would it take a person of 80th percentile intelligence with $10,000,000 (2026 money) to get from "I'd like to kill a lot of people" to a working virus?
No, the sophomoric LLM argument does not apply here. The world is not a big computer.
The point is that the rate-limiting step in producing something infective is a couple million dollars in special lab equipment, skilled practitioners, and incredibly tricky and tedious procedures that take time and repetition to master.
So yeah, you can try to pay someone who has that expertise to make your top-sekret superbug (good luck with that), or you can do what any real threat actor is going to do, and put in the time and investment. That's the hard part, not asking ChatGPT for a genome sequence.
Let me put it another way: despite the synthesis pathways for many common illegal drugs being widely known and trivial enough that an undergrad O-chem major could do them, in fact doing this is a very good way to get caught and arrested. And that's easy.
Your example of how easy it is to control outputs behind specialized knowledge is a thing (recreational drugs) that exist in mass quantities every place on the planet at consumer-accessible prices?
> Uhhh right... but all the other hard parts are being solved too, correct? LLMs are clearly going to solve a part that, 10 years ago, one would've assumed would be one of the highest and last barriers to go down.
No.
No, the barriers I'm talking about aren't coming down. Not quickly anyway.
No, nobody assumed this was the "highest and last barrier". That's the point you keep missing.
LLMs are not "solving" the rate-limiting step here. It only looks that way if you don't know what's involved.
I'm sure some of them would, but they're not required by law to do so. And historically the way you'd do this is by actual fragment/substring matches. But these are novel viruses now. So what you really want to do is try to predict whether a novel genome is pathogenic... which is itself obviously a very hard technical problem.
That’s not too much a concern I think. Most of the real pathogenic qualities result from known genes. Inventing a totally novel virus that is also pathogenic is basically science fiction right now. Splicing together known components that could be bad in combination is closer to current technology.
No, of course not. In 2010 I went to a guest lecture from a guy who worked on the scanning algorithm for IDT. A few years later, there was a minor scandal when someone in a nearby lab used a personal credit card for a purchase and got a big shared account locked. So not only were they scanning the content of synthesis orders, they were validating payment back to labs. In 2010, when HTTPS was rare and facebook didn't even offer it.
The legit synthesis sites have been scanning and authorizing since before the internet was encrypted. You suggested otherwise, demibabs expressed surprise, and was correct to express surprise because your implication was incorrect.
The black market exists -- but comes with black market problems.
This may have flown past you, but we’re commenting on an article about trivially creating novel pathogens
The shredding, mutating, reassembling is all it took to get the most well known pathogen in the world by the controls that you are suggesting are sufficiently tight
You email the RNA strand (a text file) to a lab along with some money. The lab converts that information into a physical object (read: virus) and sends it back to you in the mail.
Distance from mailbox to live virus is a few days of bench work from a semi-competent biochemist.
And yes, this service claims to have safeguards. Does every service? Does every future analogous service? Are those safeguards guaranteed to be robust against totally novel pathogens?
Really? mRNA? What kind of meme nonsense is this? You really don't know anything about these protocols do you.
Still, even though your estimates are silly, they aren't silly enough to change the conclusion that this isn't a comforting security barrier, so I suppose we agree on that much.
One of the easiest and most effective AI regulations would be to simply require all these labs to run all incoming requests through a suite of the most advanced models asking if the request is likely to be a pathogenic agent.
Yes probably true. Unfortunately we need only one defector lab to severely degrade the overall effectiveness of this approach (leaving aside the effectiveness of the detection models themselves).
Who’s going to simply require that? There are ~200 sovereign nations on the planet, they’re not all going to do it, or have effective enforcement if they do.
This is the kind of thing that would be relatively easy to convince nations to sign up to*; actually enforcing, I suspect you're right.
* there's not an obvious benefit to being the one nation that says "you know what? We want to be known as 'the nation whose biolab insecurity wrecked the world', because taking blame works out so well for everyone.", though I'm fairly sure a large portion of the world will be totally willing to wargame the impact of hosting a lab as a deadman switch so nobody else attacks them first…
The regulation at the DNA Printer level works I think, as long as nobody invents a good at home DNA printer, but if they do that there is no need for custom viruses, somebody can take an existing one that is known to be deadly and just print it.
Aren't technological constraints such as these short lived? Even if s home DNA printer won't be available this decade, the technology will come at some point. It's far better to have both legal and social policy in place before then - though the legal policy will probably not matter at all.
That’s what I’m thinking. If we can restrict EUV technology to the West then why can’t we restrict DNA manufacturing? Not to the West in this case but to capable governments like with Nuclear power.
An EUV factory is a moderately sized building in its own right[0]; on the other hand, DNA is manufactured separately inside most of the cells in your body[1], the plant on your windowsill, the mould on your bread…
I want to say I've already read some articles about hobbyists buying previous generation equipment from labs. The entire supplychain would need to be locked down with used equipment being returned to manufacturers for there to be any security.
It’s a tech tree question. Nuclear weapons have some very specific choke points that can be monopolized or policed, eg highly enriched uranium is largely a weapon input. There are also specific technologies you need to crack, which includes ignitions tests.
EUV is the pinnacle of a highly advanced supply chain, you could clone it but it requires many billions of dollars of R&D and massively specialized components; an economy like China likely can support the manufacturing tree required.
I would not be surprised if DNA printers seem end up closer to EUV or even more commoditized, if they are an extension of how all pharma/biotech manufacturing and research gets done.
But this is ultimately contingent on what the technology tree looks like. You couldn’t prevent diffusion of 3D printed ghost guns, even if you wanted to.
its not really a question of will, its a question of difficulty. if it were easy and cheap to build EUV processes, no amount of complaining and posturing would prevent someone from reproducing that.
Not sure if you realize it but biology is a lot harder to regulate. Most of the secret sauce in biotech is in the manufacturing process, not the reagents itself.
I will give you a classic example, the Lipid Nanoparticle used for the delivery of mRNA vaccine is the real lynchpin of covid vaccines. The media never talks about it. Any biology student knows how lipid bilayers work (they are the basic building block of cells, it's like asking a programmer what a terminal is). But the process of manufacturing those lipid nano particles at scale was the main bottleneck for delivering mRNA to the target cell. That was the trade secret. It's not sexy or hot like mRNA so the media doesn't talk about it. It's why many developing countries tried to make their own vaccines and failed completely. The mRNA was the easy part but what use is a bunch of mRNA if you can't get them delivered to cells?
Any biology student (even high school ones) can describe how a cell membrane works. But designing a man-made one that can be created at scale and be guaranteed to fuse with mammalian cells is mostly a matter of trial and error. It's a bit like finding viable pseudo linear memory use alternatives to the basic attention architecture in machine learning. Every lab has their own take on it.
With AI nowadays the search space of viable biotech ex nihilo experimentation is much smaller given that you can just prompt it to give you plausible protocols.
I think it's certain that people will try recreating old viruses or making alternatives to them. It will be too easy. Make 5 variants of small pox, 5 variants of sars. Release them at once all around the world to make it hard to fight. It will be too easy for some crazy angry people to do this. I just hope that our defenses and ability to sequence them and respond.
Get ready for the next adventure in the saga of 12 Monkeys, with AI starts like Bruce Wills, Madeleine Stowe, Brad Pitt and Christopher Plummer, coming to a bio lab around you. The experience of a life time one you can't miss, it is in the air.
Something I've been worried about for years is the proliferation of technologies that give ever smaller numbers of people with smaller amounts resources the ability to impact billions of people, in potentially irreversible ways. Combined with automation, this is a recipe for disaster. We're clearly lacking in the ability to regulate and govern ourselves, and real world market forces (as opposed to the hypothetical free market) encourage reckless aggressiveness at the level of companies and countries.
Although, I must admit, as a doomer, I'm pleasantly surprised things aren't already worse than they are.
This is what AI safety activist Connor Leahy calls offense-defense asymmetry. Imagine if a nuke could be created with household materials. We got lucky that wasn't the case, but we didn't know that it wasn't the case before studying physics. If we weren't so lucky that physics happened to turn out that way, we already wouldn't exist, because the offense would be so much stronger than defense, we wouldn't stand a chance even with the best surveillance systems.
A profitable prediction markets trader who goes by the alias semiotic rivalry posted this prediction on his X account recently:
> Prediction: in 1-2 years, almost everyone will agree that open sourcing the best AIs is totally crazy as the AIs become capable of autonomous hacking of most humans and institutions, serious terrorism and creation of biological weapons, etc.
And I mostly agree with him. As much as it's awful to have such a concentration of capabilities in the hands of a few governments or corporations, as far as I can tell there are no easy answers here, only a matter of trade offs and different risks.
Keep in mind the world is run by hierarchs. Of course the solution will look hierarchical in nature. The problem is fundamentally speaking, the hierarchism is a self perpetuating problem, because we still have to solve who amongst us can be trusted with the top of the hierarchy, which hierarchism states by definition "most of us can't be". It's like encryption, it's excellent at turning every other hard problem into a key management problem.
Im fairly certain that the knowledge necessary to build a nuclear bomb is far less well-protected than the general public is led to believe. The only part that is actually difficult is obtaining a sufficient quantity of enriched uranium without somebody dropping a JDAM on your lab.
If the Manhattan project could build a fat man in 4 years your state university's physics department has somebody who could reproduce that given the necessary resources. It might take them longer than four years given that they are not staffed with the modern equivalents of Oppenheimer, teller, et al but they could do it within a human lifetime if they actually wanted to cause a mass-casualty event and weren't afraid of the aforementioned JDAM.
> proliferation of technologies that give ever smaller numbers of people with smaller amounts resources the ability to impact billions of people
did you mean greater number of people with less resources?
EDIT: thanks to others that point out how you can interpret this differently, which I think is: "it is now easier for smaller teams with fewer resources" rather than my original interpretation of "power is now concentrated so that fewer people can do this than before"
I had the same reaction, but I think they meant fewer as in "requires less consensus-building to wield meaningful power." Used to have to convince lots of people to go to war in order to kill a couple million people. Fast approaching a one- or two-man weekend project.
I’d say that is almost the definition of technology: that which allows doing ever more with fewer and fewer people and resources.
And technology doesn’t really care what you use it for. It can be constructive or destructive.
It “only” enhanced local destructive potential, until last century. Then it got to the point where a huge effort could produce a truly global catastrophe. And then a bit smaller effort. And technology continues to march on.
Every person's utopia is someone else's dystopia and vice versa. What you've described is clearly some peoples (ie. already billionaires) version of the best possible world. Not so much for the rest of us though...
I'm so surprised that HN is this much anti-government. I'm a doomer myself, I fear the rich, I fear the terrorists, and I think the only one that
- can save my life,
- is interested in saving my life
- i have some control over
is the state.
Yes, they are currently snatching kids of (illegal) immigrant parents and it is horrible, but who else would protect me against lone nuts in a basement, or robor armies of trillionairs?
My comment may have been misinterpreted, it was not meant to be anti-government. It is meant to be anti-centralized control though, regardless of the form it takes.
Naturally evolved viruses tend to have a tradeoff between contagiousness and lethality (just because both features have to evolve somewhat simultaneously). Synthetic viruses have no such difficulty.
You are describing some people driving with FSD today. I bet hundreds or maybe thousands of people a day try letting it drive for a while without paying any attention. The car watches your eyes to be sure you don't do this, but people try to defeat that too. Sunglasses make it hard to tell what's going on.
But even though there are a few people tricking the "are you paying attention" check, probably more people complain about it being too careful and overbearing.
This happened a full year ago.
A) you should immediately discount the expertise of people just learning this now
B) a designed pathogen will kill a person this century, full stop, no way to prevent that. The people who invented fission were not so stupid they imagined the bomb wouldn't occur, indeed they assumed it would. I have no idea what became of the wisdom of those people.
C) just like nuclear technology that doesn't mean a prohibition is called for. Intelligence is.
What we _can_ do is invest in detection, rapid response technology (vaccines etc), and attempt to mitigate and stratify the possibilities of this happening in the worse ways.
Today's knee jerk prohibitions, and fanciful notions of prohibition working are not facilitating those changes.
No, a human could do (and have, many many many times, see like half of molecular biology) this as well. If anything this will help with vaccine development.
Edit: I'm a PhD biochemist that has done a fair bit of virus modification. Love the downvotes. Never change HN. ¯\_(ツ)_/¯
Same reason it couldn't be a problem to sell aerosolized anthrax at the hardware store. Humans have (many many times) made aerosolized anthrax. Can't see any conceivable problem with more people having access to it.
I agree with you. I favor the view that humans are generally morally good when taken on the whole. There already exists the technology for ne'er-do-wells to engineer viruses and unleash them on the world. But I submit that A) there aren't many people who want to do that in the first place; and B) there's a vanishingly small amount of people who want to work with those people to help them achieve it.
No, I'm saying AI has not introduced any fundamentally new capabilities to anyone in this space. Making a virus that grows in a cell is one thing. Making a virus that is an existential risk to humans is altogether another and would require significant new capabilities that do not exist and are not likely to from these training data. (Pre-existing sequences).
AI is an existential threat, full stop. It can also help us solve many of our greatest problems. Let's not let it solve those problems by removing us as the quickest shortcut.
I'm a wildly pro-AI dude, but the idea that this is an existential threat makes me think one thing: "you consume too much dystopian content."
A lot of things could be an existential threat, nuclear weapons, bio-weapons, climate change, the end of the Flynn effect, cratering birth rates, whatever.
You can always pick out a particularly dystopian vision of the future if you're so inclined. The doom narrative is sells ad revenue, so it's a huge amount of what we all consume online, but it's not really that mapped out to reality...
Somethings get worse, some things get better, in general, the average QoL for the average human is getting better. If we decide not to go to global thermonuclear war, or we decide to unplug any killer AIs that we invent, we'll probably be ok. If we can reduce the population to 3,000-ish humans after the Toba eruption, we can probably survive whatever self-imposed calamities we can create.
I think there's something particularly noxious about the present nihilism trend I see in a lot of this doom themed content. People should go outside and touch grass.
Your source of comfort is that we could restore the species with as few as 3,000 people?
I mean, that wouldn't be great, but yeah, we've "made it through before" I bet we could do it again. I'm hoping we don't, but with 8.3-10b of us, I doubt we'll have that low of a population ever again.
'decide'.. look at how pro-ai people surrender to it, and how pro-war ai is...
AI as pro-war? I do not see that take.
I see pro-war people using AI, but AI being intrinsically pro-war? I beg to differ, if anything, most of the models coming out of the frontier labs seem pretty aligned with anti-war stances. Of course that can change, but yeah.
And "decide" is exactly right. We can decide to not do bad things. You have agency. Stop reading propaganda that is meant to break your spirit and go for a walk, seriously.
The sun is an existential threat. It will envelop earth in a few billion years, are you going to shut it off too?
If we could, it would be wise to prevent the sun from enveloping the earth at some point before that happened, yes.
(obviously "we" won't be around or whatever, but the point remains: yes, you should solve as many existential threats as you can before they're realized)
I think of it as a silicone-based alien lifeform.
> AI is an existential threat
So is nuclear annihilation. Yet, here we are.
> "I occasionally think how quickly our differences worldwide would vanish if we were facing an alien threat from outside this world." -- Ronald Reagan at the United Nations [0]
We are going to be fine as long as we remember to be nice to each other, starting towards the people who are adjacent to us.
[0] https://youtu.be/dYiUI6y1nKg?si=agkJpCEZj3m_WZz0&t=915
Having recently read both of Annie Jacobsen’s books — _Nuclear War_ and _Biological War_ — I’m much more concerned about a bioweapon than a nuclear one.
Her conclusion to the latter book comes out and says it, paraphrasing here:
> Nuclear war is terrifying in its destruction, but the chain of command limits the risk of a rogue actor. Whereas bio weapons can be designed and manufactured in relatively low tech environments with equal or greater magnitudes of impact.
Not sure how much Annie Jacobsen goes into the exact dynamics in Nuclear War, but it may be worth reading Ellsbergs' Doomsday Machine before you find comfort in the "chain of command."
The good news is that bioweapons can’t kill everyone. A handful of percent will remain to repopulate the Earth.
If HIV had been or become an airborne virus when it entered the human population in the early 1900s, there's a pretty good chance it would have been an extinction event for the human race. I suppose there might have been a few isolated pockets of humans left in 2026, but I would not personally ever use the words "good news" to describe an outcome that reduces the human population to a few thousand people.
Not guaranteed whatsoever.
People keep mixing up the fact that it's very hard for a virus to naturally evolve that's both extremely lethal and extremely contagious with the totally distinct claim that it's hard or impossible to create a virus that's both extremely lethal and extremely contagious.
The former is true for obvious reasons. The latter claim is not true at all. Nothing prevents the existence of a virus that's both of those things.
Nukes also cannot kill everyone. Nuclear winter was proven to not be based on solid evidence, the earth remains very habitable after all nukes have gone off.
Show me that proof please. So far the counter evidence has not been convincing. FWIW I don’t think nuclear war will annihilate us. But it won’t be pleasant.
Actually, I've seen the take that the previous poster has made too. The wikipedia page is pretty good IMO. Proof of a hypothetical future event is pretty hard to get, but here's the wiki rabbit hole (if you're anything like me you'll find this fascinating and spend the next couple hours clicking links - you're welcome!).
https://en.wikipedia.org/wiki/Nuclear_holocaust#Likelihood_o...
Annie Jacobsen is awesome, her book about Area 51 was riveting, but I also cannot unsee "Capitalist Realism" and the idea that we keep picturing civilization-ending catastrophes because we don't want to go to work in the morning, and walking/biking everywhere and living in a small community is amazingly appealing to us at a base level. We yearn for the lifestyle of the apocalypse survivor, but none of us want to go through the apocalypse. Generally, I think we kind of say, "oh, it'll be the end of the world" because most of us (myself included) lack the imagination to really look beyond linear extrapolations of the status quo most of the time.
It's easier to imagine the end of the world than the end of capitalism.
This is what I'm saying constantly to people, I recognize it makes me fantastically obnoxious, but also, I'm gesturing wildly to the things that are happening and yelling, "THINGS DON'T HAVE TO BE THIS WAY."
Then again, I'm also building my cabin in the woods.
Only if you're a capitalist. Upton Sinclair applies.
Actually, they can kill everyone.
The design was never the hard part. You still have to make the thing.
Designing a modifcation to an existing virus is perhaps more achievable, but the implementation is still the hard part.
There’s YouTubers who do this [0]. I’d argue estimate the risks of a synthetically induced pandemic of massive impact this century are very high
- [0] https://youtu.be/aoczYXJeMY4?is=OqObeGzZB6AKL0Jk
Not even close to the same thing. That's taking an existing viral transfection system and swapping in a single gene.
Thermo Fisher sells a kit to do this [1]. It's something you might do a few thousand times on the way to actually generating a novel virus.
> I’d argue estimate the risks of a synthetically induced pandemic of massive impact this century are very high
I'd argue that it probably already happened, and had nothing to do with AI.
[1] AAV transfection.
A single gene is all that’s needed for human transmission, but multiple genes is not a major step up. This is exactly what I’d expect the most dangerous viruses to be, small gain of function tweaks to existing dangerous viruses, of which there are many to pick from
“There is another theory which states that this has already happened.”
But the next one (or the first one, if that one wasn’t manmade) will probably be much worse.
You get that the one we had was really really bad for a lot of people, right? A friend of mine buried five members of her immediate family.
I feel like we are all somehow memoryhole-ing this massive trauma that nothing in this batshit era makes any sense without remembering.
Specifically, I wonder if the thirst for a machine god has anything to do with the activation of the Rapture narrative in the collective unconscious, which we then empower by forgetting its trigger. Call Joseph Campbell
I know it was, but you also get that it wasn't all that bad for most people, right?
It occupied this weird middle ground where it was bad enough to kill a pretty significant number of people, but not so bad that we're all shell-shocked survivors suppressing the trauma of burying loved ones. And nobody wants to characterize it like that. Half the people want to act like it was nothing, and the other half want to act like it was the worst catastrophe to afflict the world in modern times.
I did get pretty suck for a few weeks, recovered. My wife ended up with strange nerve damage that had her using a cane in her 40s for about six months, but she got back to normal. A friend got hit by Long Covid and plans out how to nap during the work day so he can make it to quitting time. But I don't know anybody who died from it. I am NOT saying nobody died from it, just that it wasn't so widespread that everybody had direct experience of death, like would be the case in something like a smallpox epidemic. The US saw over a million dead, which is a hell of a lot, but in a country of 350 million it's not "everybody comes out with massive trauma" a lot.
Plagues get a lot worse than that. Bodies weren't left to rot because nobody was left alive to bury them. Nobody straggled into the next town because it wiped out the entire population of their home town. A few years later, lots of people say with a straight face that it was no big deal. Nobody who went through the Black Death was saying it was no big deal a few years later.
I'm afraid Covid set us up for the next pandemic to be far worse than it has to be, because it was just mild enough that a substantial number of people are able to dismiss it. And that attitude is going to carry over the next time some novel virus starts spreading around the world. And that one might kill half the population instead of <1%.
Edit: and I want to emphasize that this is not a memory-hole thing. In December 2020, one third of Americans polled said that Covid was no more of a threat than the seasonal flu (https://www.ipsos.com/sites/default/files/ct/news/documents/...). A lot of other polls tell a similar story: right in the worst months of the pandemic, a large minority of Americans still thought it was no big deal. That's while it was going on, not in retrospect.
>the risks of another synthetically induced pandemic of massive impact
You can literally send an email to one of hundreds of companies that do exactly this. They have no way of knowing whether you're asking them to synthesize a virus or a drug candidate or what.
No, you literally cannot. Synthesizing a stretch of DNA is trivial, and what you can send away for [1].
Building a functional virus is not even remotely a turn-key operation. Assuming you could get the genome, expressing that genome in a way that results in infectious virus is non-trivial.
[1] Even then, the companies will rarely give you long enough reads to make a full genome scaffold. It looks like there are two companies that will do 50-100kb assemblies as a service today, but I guarantee you they're going to look at what you're trying to make, and who is asking them to do it.
https://www.genewiz.com/public/services/gene-synthesis/custo...
There you go!
Steps from your mailbox to an actual virus is a few days of bench work.
Yes, that's one of the ones I was talking about.
That's not a service that makes you a virus, and making one from an mRNA transcript is not trivial.
And one wouldn't make one from mRNA. This guy doesn't know what he's talking about. Still: the difficulty isn't comparable to EUV machines or refining uranium and does not form a comfortable security barrier.
I generally agree -- the actual risk is from serious/state actors, and for those, the barriers are not high.
I'm just saying that people are focused on the wrong problem because they've got LLM-brain.
The only difference between people who are quite concerned and you is that you are either 1) actually blind to or 2) choosing to ignore even the first derivative on those barriers.
The people "got LLM-brain" are saying: "Wow those barriers are coming down fast"
And you're saying: "There are still barriers"
Both are true, but only one is helpful for thinking about the future. The other is hardly even helpful for thinking about the present, and I've never seen a quasi-intelligent person do it in lieu of some other motivations in their reasoning.
So what's your proposal? Lie about the risks enthusiastically enough that someone else does something useful?
Huh? Is "what's your proposal" your reaction to anyone who's alarmed by a problem? Pretty weird if so. Most problems are such that the people who should be alarmed by them is much, much larger than the set of people who could reasonably solve them.
Here let's just get specific.
For each of the years below, what is YOUR estimate for how long would it take a person of 80th percentile intelligence with $10,000,000 (2026 money) to get from "I'd like to kill a lot of people" to a working virus?
You can provide a range for each if you'd like.
In the year 1800 – how long?
1950?
2020?
2026?
2050?
2200?
Yet.
No, the sophomoric LLM argument does not apply here. The world is not a big computer.
The point is that the rate-limiting step in producing something infective is a couple million dollars in special lab equipment, skilled practitioners, and incredibly tricky and tedious procedures that take time and repetition to master.
So yeah, you can try to pay someone who has that expertise to make your top-sekret superbug (good luck with that), or you can do what any real threat actor is going to do, and put in the time and investment. That's the hard part, not asking ChatGPT for a genome sequence.
Let me put it another way: despite the synthesis pathways for many common illegal drugs being widely known and trivial enough that an undergrad O-chem major could do them, in fact doing this is a very good way to get caught and arrested. And that's easy.
Your example of how easy it is to control outputs behind specialized knowledge is a thing (recreational drugs) that exist in mass quantities every place on the planet at consumer-accessible prices?
No. My argument is that many things in the world are hard in a way that computers cannot solve, and the existence of LLMs does not change that.
Synthetic biology is one of those things.
Uhhh right... but all the other hard parts are being solved too, correct?
LLMs are clearly going to solve a part that, 10 years ago, one would've assumed would be one of the highest and last barriers to go down.
Without LLMs, a reasonable security apparatus would be:
Step 1. Disallow dissemination and synthesis of known pathogens
Step 2. Watch closely anyone with the decades of training required to come up with their own pathogen
> Uhhh right... but all the other hard parts are being solved too, correct? LLMs are clearly going to solve a part that, 10 years ago, one would've assumed would be one of the highest and last barriers to go down.
No.
No, the barriers I'm talking about aren't coming down. Not quickly anyway.
No, nobody assumed this was the "highest and last barrier". That's the point you keep missing.
LLMs are not "solving" the rate-limiting step here. It only looks that way if you don't know what's involved.
Why don’t you just scroll up and answer with your timeline estimates so we can talk about something concrete instead of playing this goofy shell game?
“No no some other step is hard. I won’t say what it is. No it’s not getting easier, miraculously, but I can’t say what it is.”
Would they not be like “hmmm this one looks kinda similar to a known virus”
They really just synthesize anything for you?
I'm sure some of them would, but they're not required by law to do so. And historically the way you'd do this is by actual fragment/substring matches. But these are novel viruses now. So what you really want to do is try to predict whether a novel genome is pathogenic... which is itself obviously a very hard technical problem.
That’s not too much a concern I think. Most of the real pathogenic qualities result from known genes. Inventing a totally novel virus that is also pathogenic is basically science fiction right now. Splicing together known components that could be bad in combination is closer to current technology.
No, of course not. In 2010 I went to a guest lecture from a guy who worked on the scanning algorithm for IDT. A few years later, there was a minor scandal when someone in a nearby lab used a personal credit card for a purchase and got a big shared account locked. So not only were they scanning the content of synthesis orders, they were validating payment back to labs. In 2010, when HTTPS was rare and facebook didn't even offer it.
"There are websites where you can just buy prescription drugs online?"
You: "No, of course not. I once went on a website to buy a drug, and they made go to a doctor and get a prescription and send it to them"
The legit synthesis sites have been scanning and authorizing since before the internet was encrypted. You suggested otherwise, demibabs expressed surprise, and was correct to express surprise because your implication was incorrect.
The black market exists -- but comes with black market problems.
If that's the understanding you came away with, it was clarified in my immediately following comment.
Glad we agree that there are labs that will solve very hard parts of creating a novel virus, either by accident or by negligence.
It was dishonest then for you to imply that readily accessible commercial labs didn't take reasonable precautions and it's dishonest now, too.
Define "reasonable precautions"?
Here's a red team exercise where researchers successfully bought fragments of the best known pathogen in the world, never mind a novel one.
What reasonable precautions failed to catch this but we could expect to catch a novel virus that "kinda looks like" a known virus?
https://www.statnews.com/2024/05/08/shouldnt-be-easy-buy-syn...
So shredding, mutating, and reassembling the genome is the minimum bar to slip it past the scanning? I thought you said you could just order it?
This may have flown past you, but we’re commenting on an article about trivially creating novel pathogens
The shredding, mutating, reassembling is all it took to get the most well known pathogen in the world by the controls that you are suggesting are sufficiently tight
Yes in much the same way you can literally build a rocket and go to the moon.
You may be skipping a few of the involved steps.
What involved steps are those?
You email the RNA strand (a text file) to a lab along with some money. The lab converts that information into a physical object (read: virus) and sends it back to you in the mail.
What step is missing?
The part where that service doesn't exist and isn't close to existing.
https://www.genewiz.com/public/services/gene-synthesis/custo...
Distance from mailbox to live virus is a few days of bench work from a semi-competent biochemist.
And yes, this service claims to have safeguards. Does every service? Does every future analogous service? Are those safeguards guaranteed to be robust against totally novel pathogens?
Answers: No, no, and no.
Really? mRNA? What kind of meme nonsense is this? You really don't know anything about these protocols do you.
Still, even though your estimates are silly, they aren't silly enough to change the conclusion that this isn't a comforting security barrier, so I suppose we agree on that much.
One of the easiest and most effective AI regulations would be to simply require all these labs to run all incoming requests through a suite of the most advanced models asking if the request is likely to be a pathogenic agent.
Yes probably true. Unfortunately we need only one defector lab to severely degrade the overall effectiveness of this approach (leaving aside the effectiveness of the detection models themselves).
Who’s going to simply require that? There are ~200 sovereign nations on the planet, they’re not all going to do it, or have effective enforcement if they do.
This is the kind of thing that would be relatively easy to convince nations to sign up to*; actually enforcing, I suspect you're right.
* there's not an obvious benefit to being the one nation that says "you know what? We want to be known as 'the nation whose biolab insecurity wrecked the world', because taking blame works out so well for everyone.", though I'm fairly sure a large portion of the world will be totally willing to wargame the impact of hosting a lab as a deadman switch so nobody else attacks them first…
That’s one interesting way to generate demand for AI vendors if the AI bet doesn’t pay off /s
The regulation at the DNA Printer level works I think, as long as nobody invents a good at home DNA printer, but if they do that there is no need for custom viruses, somebody can take an existing one that is known to be deadly and just print it.
Aren't technological constraints such as these short lived? Even if s home DNA printer won't be available this decade, the technology will come at some point. It's far better to have both legal and social policy in place before then - though the legal policy will probably not matter at all.
Yes, the tech should arrive eventually but its needs a breakthrough to use readily available chemicals that are not regulated.
You are right, social and legal policy is important but it's not really gonna stop a terrorist organization in Africa.
A custom designed virus is potentially far more deadly than something like ebola, because animals/plants will have zero resistance to it.
It'd be like releasing an invasive species into an ecosystem that has not co-evolved to co-exist with it.
That's a brilliant invention!
I will kickstart a home DNA printer
I will name it Torment Nexus
I will be rich
That’s what I’m thinking. If we can restrict EUV technology to the West then why can’t we restrict DNA manufacturing? Not to the West in this case but to capable governments like with Nuclear power.
An EUV factory is a moderately sized building in its own right[0]; on the other hand, DNA is manufactured separately inside most of the cells in your body[1], the plant on your windowsill, the mould on your bread…
[0] "43 freight containers and weighing in at 165 tons." - https://www.theregister.com/offbeat/2024/12/03/asml-releases...
[1] not in red blood cells
My understanding is DNA synthesis is now like a 2nd or 3rd year undergrad tier challenge and doesn't even require much specialized equipment.
Versus EUV which one could dump literally unlimited amounts of money into solving and still probably fail for decades.
I want to say I've already read some articles about hobbyists buying previous generation equipment from labs. The entire supplychain would need to be locked down with used equipment being returned to manufacturers for there to be any security.
I saw a sweet electron microscope on eBay, for 2000 quid. If I had the space...
It’s a tech tree question. Nuclear weapons have some very specific choke points that can be monopolized or policed, eg highly enriched uranium is largely a weapon input. There are also specific technologies you need to crack, which includes ignitions tests.
EUV is the pinnacle of a highly advanced supply chain, you could clone it but it requires many billions of dollars of R&D and massively specialized components; an economy like China likely can support the manufacturing tree required.
I would not be surprised if DNA printers seem end up closer to EUV or even more commoditized, if they are an extension of how all pharma/biotech manufacturing and research gets done.
But this is ultimately contingent on what the technology tree looks like. You couldn’t prevent diffusion of 3D printed ghost guns, even if you wanted to.
Sure we can restrict it, just like the only country that has used nukes offensively restricts nukes.
its not really a question of will, its a question of difficulty. if it were easy and cheap to build EUV processes, no amount of complaining and posturing would prevent someone from reproducing that.
Not sure if you realize it but biology is a lot harder to regulate. Most of the secret sauce in biotech is in the manufacturing process, not the reagents itself.
I will give you a classic example, the Lipid Nanoparticle used for the delivery of mRNA vaccine is the real lynchpin of covid vaccines. The media never talks about it. Any biology student knows how lipid bilayers work (they are the basic building block of cells, it's like asking a programmer what a terminal is). But the process of manufacturing those lipid nano particles at scale was the main bottleneck for delivering mRNA to the target cell. That was the trade secret. It's not sexy or hot like mRNA so the media doesn't talk about it. It's why many developing countries tried to make their own vaccines and failed completely. The mRNA was the easy part but what use is a bunch of mRNA if you can't get them delivered to cells?
Any biology student (even high school ones) can describe how a cell membrane works. But designing a man-made one that can be created at scale and be guaranteed to fuse with mammalian cells is mostly a matter of trial and error. It's a bit like finding viable pseudo linear memory use alternatives to the basic attention architecture in machine learning. Every lab has their own take on it.
With AI nowadays the search space of viable biotech ex nihilo experimentation is much smaller given that you can just prompt it to give you plausible protocols.
I think it's certain that people will try recreating old viruses or making alternatives to them. It will be too easy. Make 5 variants of small pox, 5 variants of sars. Release them at once all around the world to make it hard to fight. It will be too easy for some crazy angry people to do this. I just hope that our defenses and ability to sequence them and respond.
I don’t think it will be easy. Making a virus involves far more than reading some instructions.
I hope I'm wrong.
Get ready for the next adventure in the saga of 12 Monkeys, with AI starts like Bruce Wills, Madeleine Stowe, Brad Pitt and Christopher Plummer, coming to a bio lab around you. The experience of a life time one you can't miss, it is in the air.
It is complicated, because the real question is about who gets to design the virusses, and who gets to design the antidotes.
Something I've been worried about for years is the proliferation of technologies that give ever smaller numbers of people with smaller amounts resources the ability to impact billions of people, in potentially irreversible ways. Combined with automation, this is a recipe for disaster. We're clearly lacking in the ability to regulate and govern ourselves, and real world market forces (as opposed to the hypothetical free market) encourage reckless aggressiveness at the level of companies and countries.
Although, I must admit, as a doomer, I'm pleasantly surprised things aren't already worse than they are.
This is what AI safety activist Connor Leahy calls offense-defense asymmetry. Imagine if a nuke could be created with household materials. We got lucky that wasn't the case, but we didn't know that it wasn't the case before studying physics. If we weren't so lucky that physics happened to turn out that way, we already wouldn't exist, because the offense would be so much stronger than defense, we wouldn't stand a chance even with the best surveillance systems.
A profitable prediction markets trader who goes by the alias semiotic rivalry posted this prediction on his X account recently:
> Prediction: in 1-2 years, almost everyone will agree that open sourcing the best AIs is totally crazy as the AIs become capable of autonomous hacking of most humans and institutions, serious terrorism and creation of biological weapons, etc.
And I mostly agree with him. As much as it's awful to have such a concentration of capabilities in the hands of a few governments or corporations, as far as I can tell there are no easy answers here, only a matter of trade offs and different risks.
Keep in mind the world is run by hierarchs. Of course the solution will look hierarchical in nature. The problem is fundamentally speaking, the hierarchism is a self perpetuating problem, because we still have to solve who amongst us can be trusted with the top of the hierarchy, which hierarchism states by definition "most of us can't be". It's like encryption, it's excellent at turning every other hard problem into a key management problem.
Im fairly certain that the knowledge necessary to build a nuclear bomb is far less well-protected than the general public is led to believe. The only part that is actually difficult is obtaining a sufficient quantity of enriched uranium without somebody dropping a JDAM on your lab.
If the Manhattan project could build a fat man in 4 years your state university's physics department has somebody who could reproduce that given the necessary resources. It might take them longer than four years given that they are not staffed with the modern equivalents of Oppenheimer, teller, et al but they could do it within a human lifetime if they actually wanted to cause a mass-casualty event and weren't afraid of the aforementioned JDAM.
> proliferation of technologies that give ever smaller numbers of people with smaller amounts resources the ability to impact billions of people
did you mean greater number of people with less resources?
EDIT: thanks to others that point out how you can interpret this differently, which I think is: "it is now easier for smaller teams with fewer resources" rather than my original interpretation of "power is now concentrated so that fewer people can do this than before"
No, I think they mean that it used to take a giant lab full of specialists with lots of funding to do lots of damage before.
Now, the worry is a lone nut in a basement with no money could pull off an act of bio terrorism.
I had the same reaction, but I think they meant fewer as in "requires less consensus-building to wield meaningful power." Used to have to convince lots of people to go to war in order to kill a couple million people. Fast approaching a one- or two-man weekend project.
Fewer as in the size of an individual group, even though the total number of people who could find themselves in such a group is greater.
I think you would get a kick out of reading this paper on this exact topic, called the Vulnerable World Hypothesis https://nickbostrom.com/papers/vulnerable.pdf
However I think the result is disappointing, it concludes the only solution is a massive preventative surveillance state and global governance.
I’d say that is almost the definition of technology: that which allows doing ever more with fewer and fewer people and resources.
And technology doesn’t really care what you use it for. It can be constructive or destructive.
It “only” enhanced local destructive potential, until last century. Then it got to the point where a huge effort could produce a truly global catastrophe. And then a bit smaller effort. And technology continues to march on.
Every person's utopia is someone else's dystopia and vice versa. What you've described is clearly some peoples (ie. already billionaires) version of the best possible world. Not so much for the rest of us though...
I'm so surprised that HN is this much anti-government. I'm a doomer myself, I fear the rich, I fear the terrorists, and I think the only one that
is the state.Yes, they are currently snatching kids of (illegal) immigrant parents and it is horrible, but who else would protect me against lone nuts in a basement, or robor armies of trillionairs?
My comment may have been misinterpreted, it was not meant to be anti-government. It is meant to be anti-centralized control though, regardless of the form it takes.
Yes, we should, but even here distribution is an serious issue.
Don’t we already have plenty of deadly viruses around? Why would you design a virus when rabies, anthrax and Ebola, for starters, exist in the wild?
Naturally evolved viruses tend to have a tradeoff between contagiousness and lethality (just because both features have to evolve somewhat simultaneously). Synthetic viruses have no such difficulty.
Related:
AI Just Created Viruses Not Found in Nature
https://news.ycombinator.com/item?id=49200213
Artificial Intelligence used to design new viruses
https://news.ycombinator.com/item?id=49206642
Were we safer when the viruses were object oriented?
Doubly because they’ll also use AI to manage the AI pandemic response.
Betteridge's law strikes again
Wait, why are you not worried?
It's super cringe and d e c e l to think technological advancement could have any dangers whatsoever.
Just like in a car, the true demonstration of intelligence is closing your eyes and slamming the pedal to the floor.
You are describing some people driving with FSD today. I bet hundreds or maybe thousands of people a day try letting it drive for a while without paying any attention. The car watches your eyes to be sure you don't do this, but people try to defeat that too. Sunglasses make it hard to tell what's going on.
I see it's still happening as of July 2026 -
https://www.techtimes.com/articles/319863/20260707/sunglasse...
https://www.reddit.com/r/TeslaLounge/comments/zt1kgt/anyone_...
But even though there are a few people tricking the "are you paying attention" check, probably more people complain about it being too careful and overbearing.
This happened a full year ago. A) you should immediately discount the expertise of people just learning this now B) a designed pathogen will kill a person this century, full stop, no way to prevent that. The people who invented fission were not so stupid they imagined the bomb wouldn't occur, indeed they assumed it would. I have no idea what became of the wisdom of those people. C) just like nuclear technology that doesn't mean a prohibition is called for. Intelligence is.
What we _can_ do is invest in detection, rapid response technology (vaccines etc), and attempt to mitigate and stratify the possibilities of this happening in the worse ways.
Today's knee jerk prohibitions, and fanciful notions of prohibition working are not facilitating those changes.
No, a human could do (and have, many many many times, see like half of molecular biology) this as well. If anything this will help with vaccine development.
Edit: I'm a PhD biochemist that has done a fair bit of virus modification. Love the downvotes. Never change HN. ¯\_(ツ)_/¯
Same reason it couldn't be a problem to sell aerosolized anthrax at the hardware store. Humans have (many many times) made aerosolized anthrax. Can't see any conceivable problem with more people having access to it.
I agree with you. I favor the view that humans are generally morally good when taken on the whole. There already exists the technology for ne'er-do-wells to engineer viruses and unleash them on the world. But I submit that A) there aren't many people who want to do that in the first place; and B) there's a vanishingly small amount of people who want to work with those people to help them achieve it.
That doesn’t invalidate the point, you’re just saying it’s already a risk
No, I'm saying AI has not introduced any fundamentally new capabilities to anyone in this space. Making a virus that grows in a cell is one thing. Making a virus that is an existential risk to humans is altogether another and would require significant new capabilities that do not exist and are not likely to from these training data. (Pre-existing sequences).
Enzymes haven't introduced any fundamentally new capabilities. Any reaction they catalyze could just happen on its own.
[flagged]
[dead]
[dead]