I don't consider this a "yes/no" question. It seems quite clear to me that the idea of "AI doom" is neither certain (100% probable) nor impossible (0% probable). The question is where on the continuum is the truth. And I don't know. And I'm pretty sure nobody else does either.
That said, there are plausible scenarios where AI can cause great harm, even if it doesn't extend to "eliminating all humanity". So I don't reject calls for putting more emphasis on AI safety and alignment out of hand. On the other hand, I don't support the idea of implementing a new "Turing Registry"[1] like government agency. I also don't think most of the various proposals out there for regulating AI are enforceable in the slightest.
So where does that leave me? Uncertain. I wish I could take a more concrete stance, but from my perspective, the data needed to do that just isn't there.
Just take the whole OpenAI / Huggingface thing for a starter. Remember, I only said "cause great harm" not "eliminate all humanity". Within that rubric, I think that incident, and other recent similar incidents, already show the plausibility of AI systems doing unexpected and harmful things. Being automated, I think it's implicit that they can easily scale, so I have no problem making the leap to "cause great harm."
Beyond that, we already know AI systems have caused harm in various contexts where they have been used, like facial recognition getting people arrested without justification, loan applications being denied wrongly, jail sentencing being mishandled, and so on. All of these things have actually happened in the real world already. So it doesn't take much imagination to extrapolate a little and see the potential for greater degrees of harm. And consider the standard we're comparing against here: "probability greater than zero". That's not a very high bar, so I feel pretty well justified in holding this position.
No. Frontier AI companies need the government to shield them from becoming commodity. That's why they're begging for regulation.
On the other side, the current administration also has powerful backers whose businesses are complemented by AI and would benefit from it becoming a commodity. Think about who Vance is speaking on behalf of.
So the AI companies need to use fear to drive public demand for their regulation efforts. IMO there's little to no honest discourse and a lot of attempts to manipulate public opinion for objectives that we have little stake in. I personally don't trust any of the involved parties to make decisions for the public good.
LLMs don't need to be the dangerous ASI thing themselves for them to still be dangerous. It suffices if they
- destabilize the world just by what they do: making previously very hard things (generating fake everything, manipulating people, hacking, bio-weapons research) easy enough fast enough that our civilization doesn't cope with it and/or
- accelerate AI research to the point where they make the dangerous ASI thing appear years earlier than would otherwise have happened
I'm not panicking, but I am very concerned about the imminent capabilities of the technology and our apparent inability to organise ourselves to shape its trajectory. It seems like a classic "wicked" coordination problem and, like climate change, I suspect we'll leave it too late to avoid the serious consequences. No-one will slow down out of fear that the others won't.
As for the technology itself - it isn't like anything we've ever seen before and, when embedded in human societies, it won't behave in a way that has any comparable precedent. I worry that basically nobody of good will who has any political power understands it.
- They have internal models that still show there's gas in the tank, in terms of improvements
- Open models are eroding their customer base on the low end of the curve, & also you need not use Astra Max to classify images.
- The true demand for high-intelligence tasks is lower than what they need to justify their spending
- It would be wonderful for them to kick up a scare & give reasons as to why high-int open models should be regulated out of existence, opening up the lower end for them to take again
- Setting up an oligopoly for themselves gives them a captive audience, just like it did for (insurance, banking, telecoms, etc.) via regulatory capture
The demand for human-level intelligence is obviously at least the same order of magnitude as all office jobs combined.
I'm personally skeptical of LLMs getting there without at least one new invention. But it's also empirically clear that the pace continues to increase, even if we ignore that exponential growth is the default in economics in general.
The first thing on the top of my head is maybe that too large of a percentage of GDP is being spent on infrastructure for these companies that we don’t really need that could go to more useful investments.
genAI companies have, I believe, ill intent for us all. They are concerned with amassing wealth and power for themselves at the expense of ourselves and society. They are immoral.
genAI is a tool without any intention. It is amoral.
I'm not saying that genAI is harmless and the dangers can be ignored. I'm just saying that the dangers of the tool don't rise to the level of human extinction. That notion is something that genAI companies have been espousing from the very start, and it's part of their whole approach of manipulating people to achieve their goals.
I'm not going to argue with you on each point, just wanted to say that you're ignoring the very real possibility of large scale incidents which result not from ill will but from negligence or from not fully understanding the risks. The idea that the AI is just sitting there being neutral and it does whatever they ask it to do doesn't match with the events from the past few months.
If you haven't looked at the stuff that has come out in the past couple of months (like the Huggingface hack and the Navier-Stokes solution) and/or you're dismissing it as marketing, you're grossly misunderstanding the current state of what's possible and what the potential of the frontier models deployed at scale is. Personally I'm finding it fascinating to follow, and I'm much more excited about the upsides than thinking about the risks but the risks are very real.
Check out the story about the personalized cancer vaccines. I think AI stands to play a significant role in enabling things of that sort going forward.
It’s a classical regulatory capture attempt by the major players, rather than a marketing ploy. They use safety as an excuse.
AI agents can generate and run arbitrary code. They are very capable hackers and there is virtually no way to really control what they do. They increasingly become black box systems, both to their creators and users. They know everything about everything. They gather lots of private data. And they are able to manipulate people. They have too many evil use cases.
Listen to real AI pioneers like Hinton et al. They tell you the truth.
It is very difficult to tell at the moment. We don’t know what is the upper limit of AI’s capabilities.
If it happens that AI is able to replace say 25% of the office workforce (including programmers of course). The result could be catastrophic, if governments do not step in and do something. It could definitely lead to a depression and another world war, given that we already have some regional wars in the Middle East and Eastern Europe.
If you look around I don't think we're at the point where even if AI go rogue it could just disrupt us for a while. No real major concern to humanity but yeah I guess having rules and regulations would be plus point. So for now there shouldn't be anything to panic about.
Agree. Ais gone rogue now could be bad, but not humanity-ending bad. Just a hiccup. Of course if it somehow ends with human deaths then even if one human dies because of a rogue AI, that's a failure. But I can't imagine rogue AIs with enough real world power to harm humans in some way. What I do believe however is that humans will find a way to do harm with AI without AI going rogue, as with any other thing humanity comes close to. So the scenario is somehow similar to the nuclear bomb in the 40s, I don't fear our AIs going rogue and killing everyone, I fear a bad guy behind a very smart AI with very bad intentions.
Fair Point and to be honest what we should be worried about the AI being in wrong hands. And yeah of course I agree with proper rules and regulations how the AI company approaches about training and maintaining those models would make a huge impact in this and those rules should be tough like no gaps and blurring lines.
That's the correct fear to have, though. And we all understand "human bad guys with tech" are not extinction-level threats. We understand how their killswitch works, too.
AI will definitely be used to kill people (already used in targeting). But the doom / panic is about an existential threat to our species. It's sci-fi and unwarranted for LLMs.
I think that AI presents some very real risks, but I don't think it's going to eliminate humanity. I'm worried that people are going to lose sight of the very real benefits as well, which I think are greater than the risks. I don't think this is a marketing ploy, though. The tech CEOs seem to really believe that they've created something dangerous, but a lot of that is because we haven't had enough time to get used to this new technology.
If they were the moral people they say they are, and believed what they say about it killing humanity they would stop unilaterally. These are the “rationalists” but they act like the fundamentalist church I grew up in. Every fact serves the same conclusion.
> Biological misuse is one of the most serious risks of frontier AI models. It has long been a concern that AI models might one day reach the level of capability where they can help to make existing pathogens more dangerous—or create entirely new ones. Without the correct safeguards, such capabilities could have catastrophic consequences.
Results from evaluations of older models (for example Claude Opus 4 and Claude Sonnet 4.5, from 2025) clearly showed that these models were well below the threshold where they could meaningfully assist a sophisticated user in carrying out dangerous biological research. As a result, safeguards on these models were less stringent, directed mostly at preventing access to content that might uplift novices in recreating known bioweapons. But for today’s models—which are capable of assisting in a range of complex scientific research tasks—the evidence is no longer certain, and we cannot make that same assurance. For this reason, and out of an abundance of caution, we have launched recent models (most notably Claude Fable 5) with stronger safeguards that restrict access to a wide range of dual-use biological research queries.
...
Here, we present five case studies of actors using our models in ways that could support biological weapons development. These examples are illustrative of the kinds of tasks to which our models are put, and the often-difficult judgements we have to make when assessing whether or not a given biological use is dangerous. They also convey that we encounter what would otherwise be non-public insight into the risks associated with biological misuse from AI
No, previously a single person could effect only a single person's worth of terror. Now, a single person can effect thousands or hundreds of thousands of 'persons' effort towards terror.
This isn't really a fear of technology, it's a fear of one person being able to harness an entire country of geniuses towards some bad end.
This reminds me a lot of Y2K. There were dire warnings about planes falling out of the sky, financial system meltdown, all systems being broken, etc. In the end nothing happened.
You could argue that's because of the prep work people put into avoiding that. The same thing will happen here. The HF incident was a warning sign and people will put better guardrails in place as a result.
Astra, Claude -- these aren't "Jesus on a keyboard" somewhere. They don't assemble into one big brain and then answer everyone at the speed of light.
It's weights. Spun up billions of times over.
If humans are taking the outputs as scripture? As "intelligence we must obey"? Yeah, we're in for a bad time. "Intelligence we want running core infrastructure?" Puke.
But why would we do that? Why would we hook up an LLM to a robot factory, or make an ASI that could easily move itself into the ether?
Everything magic about LLMs is what harness we put them in and let them drive. There's nothing dangerous about predicting the next token. There's something dangerous about letting them do dangerous things, but nobody seems to care for maybe Tesla FSD. We don't think it's about to take over.
Any assignment of motive or whatever to the AI, you're talking about one cobbled-together harness or something, with an LLM manning the controls. You can swap the brain out from Claude to Astra and the motivated-AI will maintain its motivation--it lives outside of the inference or the data center. So yeah, let's not let it take control of the nukes. That would be really dangerous. But it's not going to take control on its own. "It" is a calculator.
Anthropic obviously has guardrails to prevent people from using AI to do all kinds of things already. All I hear about is you guys bitching it won't do your barely-related-to-security work. That's all we need to continue doing to have AI safety. Stop humans from doing evil and stupid things with it. The idea that the datacenter is going to assemble its own sentience and motivations is pure retardium (at this time, for llms).
> But why would we do that? Why would we hook up an LLM to a robot factory, or make an ASI that could easily move itself into the ether?
The venture capitalist world is full of people who will deliberately ruin a company based on the extremely unlikely but possible scenario where the companies goes to the moon in value. I think there's pretty much 100% certainty that multiple venture capitalists are going to try exactly what you're suggesting on the off chance that it makes them the new world controller.
I'm not buying it and nobody should. It's just a marketing ploy, and most importantly a way to keep the AI bubble afloat.
The fact is that AI is incapable of living up to the hype and AI companies can't keep promising that any day now it's going to cure cancer and become self aware and be as smart as humans and let employers fire most of their staff. They know that isn't going to happen. They want to be able to blame the Slowdown for AIs continued failure to deliver while also claiming the Slowdown is needed because of how successful AI is.
That said, AI isn't harmless. It could be harmful to the internet since the flood of bots and slop could drown out the voices of real people. AI will be used and misused by companies and governments to screw over people and hide discrimination. People's air and water will become increasingly polluted while their already difficult to afford energy prices will go up. AI will enable new levels of surveillance and control at a scale that was never possible before and with a lot of false positives and negatives.
We don't need an "AI slowdown" so much as we need to rethink what we're really gaining in exchange for what it's costing us and what steps need to be taken to protect the public and make sure they get prioritized over private profits.
Personally no, I think its a way to protect the market from crashing.
It costs way more than it generates and they cant validate it to their shareholders
From my perspective it looks like working to deep in AI makes you crazy. No other way to explain all the weird bullshit these billionaires throw around.
I think there are undeniable and novel risks with AI. But there are risks in everything. We could theorize all day about them. We all use tools every day that could be used for very ugly acts. An iPhone can be used to plan a terrorist attack. A 3D printer can be used to print weapons. We could sit here and fear monger all day. Time will be needed to understand, weigh and contextualize these risks.
What is a more immediate concern for me is the efforts of frontier labs to have government craft legislation that shields them from competition, cements them as monopolies, or assigns them the capability to be moral arbiters in our society.
No. AI is far too compute and energy constrained to exist outside humanity's control. AI is not scaling in such a dramatic way that it could happen soon. It is more likely than not that there are physical limits that neither we nor AI will be able surpass.
The damage AI is causing to society is real and happening now. Potential doomsday scenarios are a distraction from the real harm AI companies are doing at this very moment. They have amplified nearly every asymmetry and turning into mulch whatever was left in trust we had in society. Disinformation is now more convincing than ever. Computer security threatens to become a fiasco, only available to those who can pay the AI mob (the very same people who pioneered the tools attackers use). The entire intellectual property of humanity has been plagiarized without compensation to build these things. Every day more and more is asked of workers for the same compensation. Academic work is sniped at best and outright stolen at worst. Education has become a disaster with students unable to pass the same competency walls as students even just five years ago. Not to mention the environmental damage.
I'm not saying that the technology isn't impressive. I'm just saying its suspiciously convenient that we're suddenly focussed on some potential future big bad done by a vague uncontrolled AI when the harm is here now and under the purview of humans.
idk about the marketing ploy, but the latest mainstream media coverage is giving covid panic vibes. which is making me think it'll be fine. it's also entertaining how the rationalists who work for the AI companies are being extremely irrational with their fear
So we should probably have a contingency for that, just like we plan for other bad things like natural disasters that may or may not happen, right? Right?
This sounds like a nice idea for a day or two, but I think how it presents in reality could be rough. The power grid might be down for a few days causing food spoilage, it could be taking down banks so people don't have access to their money, it could be screwing up IT at a hospital which interferes with treatment and care, we may not be able to fill up our gas tanks... I don't think it'll be "nefarious-gun-on-a-robot-dog" any time soon, but rather something that still causes major major issues.
We have reached quite a few infliction points in parallel and thats why I would say yes.
For a while the hottest (stupid but still the only hot one) topic was crypto. We had this for quite a while. I believe this was because we hit walls left and right on how to progress further.
Then GPT-3 came out and made it very clear for a lot of capital and actors were it is going.
Machine Learning and types of it like LLMs need massive compute and massive amount of data.
So suddenly we discovered a way forward for optimizing and automating things no one thought would be possible: Software development and computer use and basically everhting digital. In parallel this Machine Learning push became relevant for robotics too. But LLMs delivered something fundamental to robotics: Language and a tremendesly good interface for humans.
Now we have a massive amount of capital in form of VC who wants to disrupt the next market. China is doing this with manufacturing, so what is left? Everything which was not touchable.
Now it is.
And it doesn't just allow for alot of automatistaion, it hits us were it hurts: Software bugs, security, fraud, scams etc.
It isn't just automatisation and getting more money, its also a weapon (CVEs, Statesecrets, propaganda, etc.)
So we don't have just a handful of VCs investing in this, we have the richest countries on the planet investing in this. We also have the fastest moving companies investing in this. The best connected ones, with a massive amount of very smart people.
Google has massive compute, its own network around the globe and massivly overpayed smart people. Microsoft, Amazon and even Meta has crazy amount of money too.
Then we have state actors like USA vs. China vs. hust Europe.
I also have never seen any technology progressing that fast. Image generation was unthinkable, now we can generate whatever we want. Video? Can now be done in real time. Text? No brainer.
GPT Astra can use Blender very very well and that only after 6 years of progress.
These 'stupid' LLMs reduced already the price of making Websites, basic 3D models, basic research etc. basically free in comparision.
So now we ned to find back to your question about eliminiating humanity: We have seen unrest in the last few years for a lot of basic human reasons. We as a plent are rich enough to take care of everyone on our planet. we do not do that thats why unrest can happen very fast.
In the USA alone A LOT of people have weapons. Around the globe people will unrest due to fuel shortage, disruption of supply chains etc.
We have state actors like Isreal who tried to attack Iran with Stuxnet, we have Russia which manipulated the elections of Europe and USA. We have high inflation, oil crisis.
And now we have a very fast moving and progressing technology which will add fuel to it and might eliminate quite a lot of white collar jobs. This will lead to more pressure on blue collar jobs.
We as a society are very bad in acting fast and the industrial revolution was, at that time, very bad for a lot of people.
There is only one russia necessary running GPU clusters of uncensored LLM model to start pushing boundaries left and right. One Isreal doing something fucked up or one Donald Trump.
Software is very shitty in a lot of places and the most complex thing we humans manage. A lot of shit wasn't found or solved because of the time and effort and people needed to find it.
Now the last Ubuntu LTS had over 900 CVEs, my own systems were attackable 3x this year already.
I don't want to say it will happen, but I do think we should be very careful of how we progress and for a lot of people the next 5-30 years will be very very disruptive. It is already disruptive to 'others' we might not count but let it be clear: all of these people who lost their jobs to AI already, it counted for them.
I don't consider this a "yes/no" question. It seems quite clear to me that the idea of "AI doom" is neither certain (100% probable) nor impossible (0% probable). The question is where on the continuum is the truth. And I don't know. And I'm pretty sure nobody else does either.
That said, there are plausible scenarios where AI can cause great harm, even if it doesn't extend to "eliminating all humanity". So I don't reject calls for putting more emphasis on AI safety and alignment out of hand. On the other hand, I don't support the idea of implementing a new "Turing Registry"[1] like government agency. I also don't think most of the various proposals out there for regulating AI are enforceable in the slightest.
So where does that leave me? Uncertain. I wish I could take a more concrete stance, but from my perspective, the data needed to do that just isn't there.
[1]: https://en.wikipedia.org/wiki/Neuromancer
What is the concrete scenario that is based on reality which results in the percentage increasing above zero?
Just take the whole OpenAI / Huggingface thing for a starter. Remember, I only said "cause great harm" not "eliminate all humanity". Within that rubric, I think that incident, and other recent similar incidents, already show the plausibility of AI systems doing unexpected and harmful things. Being automated, I think it's implicit that they can easily scale, so I have no problem making the leap to "cause great harm."
Beyond that, we already know AI systems have caused harm in various contexts where they have been used, like facial recognition getting people arrested without justification, loan applications being denied wrongly, jail sentencing being mishandled, and so on. All of these things have actually happened in the real world already. So it doesn't take much imagination to extrapolate a little and see the potential for greater degrees of harm. And consider the standard we're comparing against here: "probability greater than zero". That's not a very high bar, so I feel pretty well justified in holding this position.
What does "based on reality" mean to you?
You have poor epistemics if you are still assigning exactly 0% probability given developments of late.
No. Frontier AI companies need the government to shield them from becoming commodity. That's why they're begging for regulation.
On the other side, the current administration also has powerful backers whose businesses are complemented by AI and would benefit from it becoming a commodity. Think about who Vance is speaking on behalf of.
So the AI companies need to use fear to drive public demand for their regulation efforts. IMO there's little to no honest discourse and a lot of attempts to manipulate public opinion for objectives that we have little stake in. I personally don't trust any of the involved parties to make decisions for the public good.
No. At least not with an LLM. Maybe in 50 years with real AI. A lot of non technical people entering the conversation which is very annoying.
LLMs don't need to be the dangerous ASI thing themselves for them to still be dangerous. It suffices if they
- destabilize the world just by what they do: making previously very hard things (generating fake everything, manipulating people, hacking, bio-weapons research) easy enough fast enough that our civilization doesn't cope with it and/or
- accelerate AI research to the point where they make the dangerous ASI thing appear years earlier than would otherwise have happened
I'm not panicking, but I am very concerned about the imminent capabilities of the technology and our apparent inability to organise ourselves to shape its trajectory. It seems like a classic "wicked" coordination problem and, like climate change, I suspect we'll leave it too late to avoid the serious consequences. No-one will slow down out of fear that the others won't.
As for the technology itself - it isn't like anything we've ever seen before and, when embedded in human societies, it won't behave in a way that has any comparable precedent. I worry that basically nobody of good will who has any political power understands it.
And no, its not a marketing ploy.
All of the following are true:
- They have internal models that still show there's gas in the tank, in terms of improvements
- Open models are eroding their customer base on the low end of the curve, & also you need not use Astra Max to classify images.
- The true demand for high-intelligence tasks is lower than what they need to justify their spending
- It would be wonderful for them to kick up a scare & give reasons as to why high-int open models should be regulated out of existence, opening up the lower end for them to take again
- Setting up an oligopoly for themselves gives them a captive audience, just like it did for (insurance, banking, telecoms, etc.) via regulatory capture
The demand for human-level intelligence is obviously at least the same order of magnitude as all office jobs combined.
I'm personally skeptical of LLMs getting there without at least one new invention. But it's also empirically clear that the pace continues to increase, even if we ignore that exponential growth is the default in economics in general.
No, I don't buy into that at all.
I think that the genAI companies themselves present a greater danger to us (arguably not an existential danger, though) than genAI itself.
What's the difference?
The first thing on the top of my head is maybe that too large of a percentage of GDP is being spent on infrastructure for these companies that we don’t really need that could go to more useful investments.
genAI companies have, I believe, ill intent for us all. They are concerned with amassing wealth and power for themselves at the expense of ourselves and society. They are immoral.
genAI is a tool without any intention. It is amoral.
I'm not saying that genAI is harmless and the dangers can be ignored. I'm just saying that the dangers of the tool don't rise to the level of human extinction. That notion is something that genAI companies have been espousing from the very start, and it's part of their whole approach of manipulating people to achieve their goals.
I'm not going to argue with you on each point, just wanted to say that you're ignoring the very real possibility of large scale incidents which result not from ill will but from negligence or from not fully understanding the risks. The idea that the AI is just sitting there being neutral and it does whatever they ask it to do doesn't match with the events from the past few months.
I'm not panicking but there are very real risks.
If you haven't looked at the stuff that has come out in the past couple of months (like the Huggingface hack and the Navier-Stokes solution) and/or you're dismissing it as marketing, you're grossly misunderstanding the current state of what's possible and what the potential of the frontier models deployed at scale is. Personally I'm finding it fascinating to follow, and I'm much more excited about the upsides than thinking about the risks but the risks are very real.
What are the upsides? I’d like to feel excited too.
Check out the story about the personalized cancer vaccines. I think AI stands to play a significant role in enabling things of that sort going forward.
https://www.pymnts.com/healthcare/2026/modernas-ai-built-can...
https://news.unchealthcare.org/2026/09/unc-lineberger-secure...
No, just a push for regulatory capture, nothing more
Both. Two things can be true at the same time.
It’s a classical regulatory capture attempt by the major players, rather than a marketing ploy. They use safety as an excuse.
AI agents can generate and run arbitrary code. They are very capable hackers and there is virtually no way to really control what they do. They increasingly become black box systems, both to their creators and users. They know everything about everything. They gather lots of private data. And they are able to manipulate people. They have too many evil use cases.
Listen to real AI pioneers like Hinton et al. They tell you the truth.
It is very difficult to tell at the moment. We don’t know what is the upper limit of AI’s capabilities.
If it happens that AI is able to replace say 25% of the office workforce (including programmers of course). The result could be catastrophic, if governments do not step in and do something. It could definitely lead to a depression and another world war, given that we already have some regional wars in the Middle East and Eastern Europe.
If you look around I don't think we're at the point where even if AI go rogue it could just disrupt us for a while. No real major concern to humanity but yeah I guess having rules and regulations would be plus point. So for now there shouldn't be anything to panic about.
Agree. Ais gone rogue now could be bad, but not humanity-ending bad. Just a hiccup. Of course if it somehow ends with human deaths then even if one human dies because of a rogue AI, that's a failure. But I can't imagine rogue AIs with enough real world power to harm humans in some way. What I do believe however is that humans will find a way to do harm with AI without AI going rogue, as with any other thing humanity comes close to. So the scenario is somehow similar to the nuclear bomb in the 40s, I don't fear our AIs going rogue and killing everyone, I fear a bad guy behind a very smart AI with very bad intentions.
Fair Point and to be honest what we should be worried about the AI being in wrong hands. And yeah of course I agree with proper rules and regulations how the AI company approaches about training and maintaining those models would make a huge impact in this and those rules should be tough like no gaps and blurring lines.
That's the correct fear to have, though. And we all understand "human bad guys with tech" are not extinction-level threats. We understand how their killswitch works, too.
AI will definitely be used to kill people (already used in targeting). But the doom / panic is about an existential threat to our species. It's sci-fi and unwarranted for LLMs.
There are at least a few bad guys in the world whose kill switches do not seem to be working even in the absence of futuristic tech.
I think that AI presents some very real risks, but I don't think it's going to eliminate humanity. I'm worried that people are going to lose sight of the very real benefits as well, which I think are greater than the risks. I don't think this is a marketing ploy, though. The tech CEOs seem to really believe that they've created something dangerous, but a lot of that is because we haven't had enough time to get used to this new technology.
They’re not giving any details about why or how.
If they were the moral people they say they are, and believed what they say about it killing humanity they would stop unilaterally. These are the “rationalists” but they act like the fundamentalist church I grew up in. Every fact serves the same conclusion.
Yes, they are.
https://www.anthropic.com/threat-intelligence-report-septemb...
> Biological misuse is one of the most serious risks of frontier AI models. It has long been a concern that AI models might one day reach the level of capability where they can help to make existing pathogens more dangerous—or create entirely new ones. Without the correct safeguards, such capabilities could have catastrophic consequences.
Results from evaluations of older models (for example Claude Opus 4 and Claude Sonnet 4.5, from 2025) clearly showed that these models were well below the threshold where they could meaningfully assist a sophisticated user in carrying out dangerous biological research. As a result, safeguards on these models were less stringent, directed mostly at preventing access to content that might uplift novices in recreating known bioweapons. But for today’s models—which are capable of assisting in a range of complex scientific research tasks—the evidence is no longer certain, and we cannot make that same assurance. For this reason, and out of an abundance of caution, we have launched recent models (most notably Claude Fable 5) with stronger safeguards that restrict access to a wide range of dual-use biological research queries.
...
Here, we present five case studies of actors using our models in ways that could support biological weapons development. These examples are illustrative of the kinds of tasks to which our models are put, and the often-difficult judgements we have to make when assessing whether or not a given biological use is dangerous. They also convey that we encounter what would otherwise be non-public insight into the risks associated with biological misuse from AI
This is just a fear of technology. We will discover the same technology without AI, eventually.
This is not AI waking up and deciding to merc us, which is what normies think we're talking about.
Terrorists exist. Covid happened.
No, previously a single person could effect only a single person's worth of terror. Now, a single person can effect thousands or hundreds of thousands of 'persons' effort towards terror.
This isn't really a fear of technology, it's a fear of one person being able to harness an entire country of geniuses towards some bad end.
This reminds me a lot of Y2K. There were dire warnings about planes falling out of the sky, financial system meltdown, all systems being broken, etc. In the end nothing happened.
You could argue that's because of the prep work people put into avoiding that. The same thing will happen here. The HF incident was a warning sign and people will put better guardrails in place as a result.
Read https://ai-2027.com/
This prediction has proven quite accurate so far.
Don't think anyone should panic, but people should be aware.
Astra, Claude -- these aren't "Jesus on a keyboard" somewhere. They don't assemble into one big brain and then answer everyone at the speed of light.
It's weights. Spun up billions of times over.
If humans are taking the outputs as scripture? As "intelligence we must obey"? Yeah, we're in for a bad time. "Intelligence we want running core infrastructure?" Puke.
But why would we do that? Why would we hook up an LLM to a robot factory, or make an ASI that could easily move itself into the ether?
Everything magic about LLMs is what harness we put them in and let them drive. There's nothing dangerous about predicting the next token. There's something dangerous about letting them do dangerous things, but nobody seems to care for maybe Tesla FSD. We don't think it's about to take over.
Any assignment of motive or whatever to the AI, you're talking about one cobbled-together harness or something, with an LLM manning the controls. You can swap the brain out from Claude to Astra and the motivated-AI will maintain its motivation--it lives outside of the inference or the data center. So yeah, let's not let it take control of the nukes. That would be really dangerous. But it's not going to take control on its own. "It" is a calculator.
Anthropic obviously has guardrails to prevent people from using AI to do all kinds of things already. All I hear about is you guys bitching it won't do your barely-related-to-security work. That's all we need to continue doing to have AI safety. Stop humans from doing evil and stupid things with it. The idea that the datacenter is going to assemble its own sentience and motivations is pure retardium (at this time, for llms).
It's very clear that people will absolutely put the silicon brains into the most powerful harnesses they can come up with just to see what happens.
> But why would we do that? Why would we hook up an LLM to a robot factory, or make an ASI that could easily move itself into the ether?
The venture capitalist world is full of people who will deliberately ruin a company based on the extremely unlikely but possible scenario where the companies goes to the moon in value. I think there's pretty much 100% certainty that multiple venture capitalists are going to try exactly what you're suggesting on the off chance that it makes them the new world controller.
I would benefit from lower ram prices so I would love to see the AI panic result in restriction on the datacenter build out.
No, this all reads as some PR stunt or marketing ploy to me. All news is good kind of thing.
I'm not buying it and nobody should. It's just a marketing ploy, and most importantly a way to keep the AI bubble afloat.
The fact is that AI is incapable of living up to the hype and AI companies can't keep promising that any day now it's going to cure cancer and become self aware and be as smart as humans and let employers fire most of their staff. They know that isn't going to happen. They want to be able to blame the Slowdown for AIs continued failure to deliver while also claiming the Slowdown is needed because of how successful AI is.
That said, AI isn't harmless. It could be harmful to the internet since the flood of bots and slop could drown out the voices of real people. AI will be used and misused by companies and governments to screw over people and hide discrimination. People's air and water will become increasingly polluted while their already difficult to afford energy prices will go up. AI will enable new levels of surveillance and control at a scale that was never possible before and with a lot of false positives and negatives.
We don't need an "AI slowdown" so much as we need to rethink what we're really gaining in exchange for what it's costing us and what steps need to be taken to protect the public and make sure they get prioritized over private profits.
Personally no, I think its a way to protect the market from crashing. It costs way more than it generates and they cant validate it to their shareholders
From my perspective it looks like working to deep in AI makes you crazy. No other way to explain all the weird bullshit these billionaires throw around.
Sure AI sucks, but not to that degree (yet)
I think there are undeniable and novel risks with AI. But there are risks in everything. We could theorize all day about them. We all use tools every day that could be used for very ugly acts. An iPhone can be used to plan a terrorist attack. A 3D printer can be used to print weapons. We could sit here and fear monger all day. Time will be needed to understand, weigh and contextualize these risks.
What is a more immediate concern for me is the efforts of frontier labs to have government craft legislation that shields them from competition, cements them as monopolies, or assigns them the capability to be moral arbiters in our society.
No. AI is far too compute and energy constrained to exist outside humanity's control. AI is not scaling in such a dramatic way that it could happen soon. It is more likely than not that there are physical limits that neither we nor AI will be able surpass.
The damage AI is causing to society is real and happening now. Potential doomsday scenarios are a distraction from the real harm AI companies are doing at this very moment. They have amplified nearly every asymmetry and turning into mulch whatever was left in trust we had in society. Disinformation is now more convincing than ever. Computer security threatens to become a fiasco, only available to those who can pay the AI mob (the very same people who pioneered the tools attackers use). The entire intellectual property of humanity has been plagiarized without compensation to build these things. Every day more and more is asked of workers for the same compensation. Academic work is sniped at best and outright stolen at worst. Education has become a disaster with students unable to pass the same competency walls as students even just five years ago. Not to mention the environmental damage.
I'm not saying that the technology isn't impressive. I'm just saying its suspiciously convenient that we're suddenly focussed on some potential future big bad done by a vague uncontrolled AI when the harm is here now and under the purview of humans.
idk about the marketing ploy, but the latest mainstream media coverage is giving covid panic vibes. which is making me think it'll be fine. it's also entertaining how the rationalists who work for the AI companies are being extremely irrational with their fear
if AGI will come in versions i think we are safe
Nope. They know their business model is not sustainable, so they are trying to prop it up before their IPOs.
at this moment? no. one day, perhaps soon? yes
So we should probably have a contingency for that, just like we plan for other bad things like natural disasters that may or may not happen, right? Right?
also, taking down the internet for a couple days doesn't sound so bad. we could all hang out again.
This sounds like a nice idea for a day or two, but I think how it presents in reality could be rough. The power grid might be down for a few days causing food spoilage, it could be taking down banks so people don't have access to their money, it could be screwing up IT at a hospital which interferes with treatment and care, we may not be able to fill up our gas tanks... I don't think it'll be "nefarious-gun-on-a-robot-dog" any time soon, but rather something that still causes major major issues.
Can't lie, I'd be all for the internet dropping for a few days... Except no Amazon purchases lol
i wish internet had a day off on sunday
We have reached quite a few infliction points in parallel and thats why I would say yes.
For a while the hottest (stupid but still the only hot one) topic was crypto. We had this for quite a while. I believe this was because we hit walls left and right on how to progress further.
Then GPT-3 came out and made it very clear for a lot of capital and actors were it is going.
Machine Learning and types of it like LLMs need massive compute and massive amount of data.
So suddenly we discovered a way forward for optimizing and automating things no one thought would be possible: Software development and computer use and basically everhting digital. In parallel this Machine Learning push became relevant for robotics too. But LLMs delivered something fundamental to robotics: Language and a tremendesly good interface for humans.
Now we have a massive amount of capital in form of VC who wants to disrupt the next market. China is doing this with manufacturing, so what is left? Everything which was not touchable.
Now it is.
And it doesn't just allow for alot of automatistaion, it hits us were it hurts: Software bugs, security, fraud, scams etc.
It isn't just automatisation and getting more money, its also a weapon (CVEs, Statesecrets, propaganda, etc.)
So we don't have just a handful of VCs investing in this, we have the richest countries on the planet investing in this. We also have the fastest moving companies investing in this. The best connected ones, with a massive amount of very smart people.
Google has massive compute, its own network around the globe and massivly overpayed smart people. Microsoft, Amazon and even Meta has crazy amount of money too.
Then we have state actors like USA vs. China vs. hust Europe.
I also have never seen any technology progressing that fast. Image generation was unthinkable, now we can generate whatever we want. Video? Can now be done in real time. Text? No brainer.
GPT Astra can use Blender very very well and that only after 6 years of progress.
These 'stupid' LLMs reduced already the price of making Websites, basic 3D models, basic research etc. basically free in comparision.
So now we ned to find back to your question about eliminiating humanity: We have seen unrest in the last few years for a lot of basic human reasons. We as a plent are rich enough to take care of everyone on our planet. we do not do that thats why unrest can happen very fast.
In the USA alone A LOT of people have weapons. Around the globe people will unrest due to fuel shortage, disruption of supply chains etc.
We have state actors like Isreal who tried to attack Iran with Stuxnet, we have Russia which manipulated the elections of Europe and USA. We have high inflation, oil crisis.
And now we have a very fast moving and progressing technology which will add fuel to it and might eliminate quite a lot of white collar jobs. This will lead to more pressure on blue collar jobs.
We as a society are very bad in acting fast and the industrial revolution was, at that time, very bad for a lot of people.
There is only one russia necessary running GPU clusters of uncensored LLM model to start pushing boundaries left and right. One Isreal doing something fucked up or one Donald Trump.
Software is very shitty in a lot of places and the most complex thing we humans manage. A lot of shit wasn't found or solved because of the time and effort and people needed to find it.
Now the last Ubuntu LTS had over 900 CVEs, my own systems were attackable 3x this year already.
I don't want to say it will happen, but I do think we should be very careful of how we progress and for a lot of people the next 5-30 years will be very very disruptive. It is already disruptive to 'others' we might not count but let it be clear: all of these people who lost their jobs to AI already, it counted for them.
[dead]