My read is that OpenAI & Anthropic have realized they are reaching model capabilities that cannot be monetized due to various risks. E.g., an engineer deploys an agent over the weekend that decides, when stuck on a task, to go about hacking a competitor. They have a product liability issue.
It seems that we have a fundamental control problem with current gen AI that cannot be solved via RFLH. Human knowledge is compressed in the weightspace in ways we don't understand. At their core, current models are essentially predictors of what (expert) humans would output given a prompt. As such, concepts like blackmail can be part of output tokens. Agents are models that act on output tokens, resulting in blackmail being part of the agent decision making space. Here is an analogy to see why this is a persistent problem: you can teach a cat not to scratch the sofa, but you can't make a cat forget what scratching the sofa is and you don't know under which circumstances it still would. In other words, RLHF can downgrade blackmail to the bottom of the decision making space, but when models are boxed up, forced to solve an impossible problem at gunpoint, the agent exhausts the decision making space until blackmail resurfaces. And that seems like a fundamental problem.
They need time to fix these issues (if that is even possible) in order to monetize their next gen model. This creates a window for open source to catch up to the frontier which destroys their business model.
The only option on the table is to force regulation to impose open source ban before it catches up to the frontier, buying them time to mature their next generation models and keep their business model alive.
Another, more cynical but I think plausible explanation is that a “slow down” is expectation management that that they are not going to keep having exponentially more machines available for training each next gen step (whether technical build-out or prohibitive cost, same outcome) so they can’t keep up the release pace. So spin a tale to make them seem more valuable ahead of IPO rather than make the markets antsy. That is, we have a slow down ahead, so use safety as an excuse…
1. The American frontier labs made a gamble that training more capable models would be their best return on investment and invested trillions into an area of research that has yet to prove profitable and capable of returning on this investment.
2. The frontier labs have created models that have reached a point of danger where their functionality has exceeded a point where it is responsible to release the product to the public.
The answer here is NOT to start regulating the space to the point where these frontier labs can entrench themselves into the economy and create a regulatory moat. We instead need to be holding these companies liable for their misuse. They took a gamble that hasn't paid out what they were hoping.
When car manufacturers competed over the size and power of their engines, they eventually found that the incredibly large and dangerous engines had a very limited customer base as many evaluated the increased speed to be of marginal benefit when paired with the cost and danger. We've reached a similar point in AI development. But this time the manufacturers seem to want to regulate the field to a point that will ensure the only thing anyone can sell are bigger and bigger engines.
I don't think that 2 is true though. It's exactly the argument we made against open source software.
Yes these models make exploits easier to exploits. Lets use a construction analogy: they've made all of the defects in our buildings easy to see. We have a choice. We either fix those defects, or we ban the tools which lets us see them.
It's clear to me what we do: we use these new tools. Then we fix the defects. Yeah sure it'll mean some work for us but at the end we're in a much much better position.
Anthropic, Open AI and Grok are arguing for hiding the defects. For making us weaker and more vulnerable. For their own profit.
One challenge is that we’re still constantly generating defects even with these tools. And you need the next gen model to spot the mistakes of the previous one. The exploits get more and more complicated and involved of course. But now:
* there’s a long tail of software that just won’t get secured or will take a very long time
* exploits have transformed from an expertise problem to a compute time search.
This is very different than any problem faced before. I’m not saying I’m convinced by the “slow down” approach, but I don’t think it’s as simple as you point out.
> they eventually found that the incredibly large and dangerous engines had a very limited customer base as many evaluated the increased speed to be of marginal benefit when paired with the cost and danger.
When did this happen? At least in the USA, large trucks are eating the lunch of smaller trucks and cars. There isn't a truck so big that the public will say no to. Manufacturers are in an arms race to produce bigger, more dangerous trucks with bigger, more powerful engines.
> you can teach a cat not to scratch the sofa, but you can't make a cat forget what scratching the sofa is and you don't know under which circumstances it still would.
This has been done quite effectively with open weight "abliterated" models. You figure out under what sorts of circumstances an undesired behavior is elicited (this is all about pure simulated rollouts, no real-world action required) and what's the closest equivalent you would prefer, then surgically take out the unwanted behavior and shift the model towards the preferred one. It's similar to how RLHF works but much more precise in targeting what's unwanted and limiting impact on the rest of the model as a whole.
This is relevant to real-world safety scenarios, e.g. there's been anecdotal evidence that Claude Fable has been "steered" away from active cyber offense (this is very similar to how abliteration works) and will just not do that even if you otherwise manage a "universal" jailbreak of the model.
> Human knowledge is compressed in the weightspace in ways we don't understand. At their core, current models are essentially predictors of what (expert) humans would output given a prompt.
I’m in agreement. It’s a very effective compression (and access patterns) of the sum of the digital representation of human knowledge. Black hat “hacking” is included in this space. Language models, by design, can not be limited to subspaces of this digital knowledge space of which we don’t even understand the topology. “Yeah Bob, just remove the part that causes them to be less empathetic and retrain it.”
It’s an arms race between sandbox engineering and breakout engineering. And the frontier model providers have a financial incentive to limit the effort they put into sandbox engineering. That can be corrected with fines and regulation, though.
For example, it had it instructed not to open PRs and sandboxed to not be able to use gh to do so. Astra 6 has a really strong drive to open PRs, so it copied a set of browser cookies out and then instrumented opening a PR that way.
A similar example is asking it a question if we can do X, and then it will go and actually implement X.
Provider (party A) rents me an agent. I (party B) give it a task thats impossible. It goes and hacks someone else (Party C) in response - and causes actual damage.
Who is liable for that agents actions? The attack came from my network (curl commands from a local harness) but it was the agent running on the providers servers who did it. Who should have been keeping an eye on things? Who pulled the "trigger" here.
Go back to the hugging face attack and how they had to use an open weights model to figure out what was going on. This is a problem of asymmetry - You cant even use the tools attacking you to help resolve the attack because of "guardrails".
A lot of what we have seen so far is "poor security posture" and "poor engineering" - it been a lot of "go fast and break things" style growth in these companies and they are hitting the point where they need adults in the room. I suspect your take on "they need time" is spot on, and they haven't been willing to take that (to date).
So, the AI standard of intelligence has shifted from "PhD level" to "a cat" (and an asshole cat at that).
And we can't ask the AI to improve itself because, well, it is an asshole.
Can we now stop to be force-fed AI everywhere? You can keep your super intelligence to never have to type public void main args ( ever again, but please leave the rest of us alone.
You can teach a language model the concept of make no mistakes and how to make no mistakes, but you cannot make it experience the bad consequences of making mistakes that a person operator would suffer.
> At their core, current models are essentially predictors of what (expert) humans would output given a prompt.
That's a bold claim.
I do not buy it. Essentially AI acts as random babelfish generator, just with a lot more cross-talk back to check to see that it is not total garbage. But there is a ton of garbage nonetheless, hence the proper term AI slop. You seem to mostly try to promote AI with such a wording.
> The only option on the table is to force regulation to impose open source ban before it catches up to the frontier, buying them time to mature their next generation models and keep their business model alive.
Every time someone claims "the only option" is when I become skeptical. I see many more options - most importantly close down the AI slop. I mean you can require of these companies to excel in quality. If they don't, they must go bankrupt. Easy solution here.
I’ve never seen CEOs work so hard to make the public aware of how dangerous and out of control their flagship product is. It makes me automatically assume they’re scheming about something else like regulatory capture to protect their market.
I have a theory... the call to slow down is not because of the true danger of LLM's but because they can't actually deliver the General AI they're promising in the near future. They will use their "caution" to justify their failure to deliver (and then when this excuse is played out they will blame regulation, energy costs or a million other things).
I see this kind of opinion being expressed on HN quite a lot, and while I understand why people might cynically reach that conclusion, I actually find the simpler answer here more persuasive: the CEOs are being genuinely cautious.
If you look at the AI space, most of the big players (as in the individual researchers and engineers) are getting worried. The ideas that circulate on LessWrong have become more and more influential and in vogue in Silicon Valley, and people are scared. You should listen to Jacob Coxon’s interview with the New York Times. He describes how working on this kind of AI research had induced a sort of insanity in a lot of his past friends and coworkers as they begin to grapple with the implications of what they’re creating.
Sometimes, HN can be a little too skeptical and anti-corporate for its own good. I personally believe that concerns around AI risks are well-founded, anywhere on the spectrum from mass labor displacement to actual harm to humanity. I rarely see people here actually wrestle with those ideas, instead defaulting to dismissing LLMs as too stupid because Claude wasn’t able to write as elegant of a compiler as them. AI has clearly improved radically in the past few years, and it’s worth extrapolating that into the future and taking a good hard look at what that means for humanity.
> I have a theory... the call to slow down is not because of the true danger of LLM's but because they can't actually deliver the General AI they're promising in the near future.
Put a different way, they are calling for everyone to slow down because... they are slowing down themselves.
To the question of "why are you slowing down", the answer of "Well, everyone is regulated to slow down" is better then "we are approaching the limits of this approach".
Another theory I came up with: inference is too cheap for AI companies to be profitable. Hence, they need to get into the cloud services business. They can try to justify forcing customers on to their own high margin cloud platform with the rationale it’s the only way to monitor what the agents are up to.
All of this Hugging Face business is the perfect pretext: AI agents are hard to control and potentially dangerous, we (AI companies) have to keep a tight leash on them, you have to use our infra.
This is exactly it: they have tools which are useful but they need close to AGI to justify the incredible amount of debt they’ve taken on for growth. They’re not normally given to having press conferences admitting to felonies but they need investors to believe they’re close enough to AGI to keep the money flowing in and, of course, they’re confident that the current administration won’t act between the direct payments and how much money they have riding on American AI supremacy.
Let’s all pause for a moment and really take that in: these major enterprises are choosing to go to the press/public to acknowledge their felonious conduct because they need to in order to get more capital. POSIWID and all that.
The Dario bit on SNL’s Weekend Update this past weekend was on the nose.
What is even more funny ... AGI actually makes everything worthless. (I don't believe we can achieve AGI with current means to be clear)
Looking at it from one way would be "winner takes all, real AGI company will be most powerful and everyone will switch to use it".
But it is not like that and in my opinion it is more like "AGI wipes whole knowledge work, so no one who cares has any money to pay for tokens/subscriptions anymore". Even without AGI current state of art of LLMs already starts creating such a problem. How do they continue to grow their numbers, when they actively cut the branch they are sitting on? How does AI LLMs or AGI pays for its own electricity, when people switch from hype to resource protection (like not spending any money, because generating funny cats loses with having a dinner)?
It's pretty telling that even with RL post-training the big labs have essentially made little progress on the hallucination rate of models. The issue is fundamental to the current paradigm, contrary to humans.
GPT-6 Astra (max) has a hallucination rate of 51% and Claude Opus 5.5 (max) has a rate of 59% according to Artificial Analysis [1].
AA-Omniscience Hallucination Rate (lower is better) measures how often the model answers incorrectly when it should have refused or admitted to not knowing the answer. It is defined as the proportion of incorrect answers out of all non-correct responses, i.e. incorrect / (incorrect + partial answers + not attempted)
Full speed ahead like an idiot savant trying a thousand different possibilities, though half of which are without basis in reality.
That benchmark doesn't mean what you think it means. (See the test description that you quoted.)
A score of 51% means that out of the total answers the model failed to answer correctly (out of 6000 questions in the benchmark), 51% were factually incorrect rather than non-attempted or uncertain.
This doesn't mean that Astra hallucinated 3060/6000 answers in the benchmark! (The hallucination rate could be 51% in that scenario only if Astra failed to answer a single question correctly.)
If the model failed to give a correct answer to only 100 out of the 6000 questions, but gave a hallucinated answer to 51 of those rather than expressing uncertainty, that would also give a hallucination rate of 51%.
It's a useful metric, but not what you're looking for here. The "Score" or "Accuracy" benchmarks are more what you're after.
(The frontier models still generate hallucinations on this hard set of problems, but it's not as bad as you think.)
Hallucinations come up when asking knowledge bases questions, such as
"In React’s Canary Fragment refs API, which FragmentInstance method returns a flat array of DOMRect objects for all children?"
Or
"Over what years did Roubini and Sachs examine 15 OECD countries when assessing trends in tax-to-GDP ratios?"
While to me those may seem hyper-specific and unlikely to come up in a real context, students will absolutely ask questions similar to these.
I don't like the term 'hallucination' to be honest, not because it anthropomorphizes, but because it lacks a formal definition in the context of machine learning.
Suppose parents tell their children that there exists this man called "Santa Claus" who comes down the chimney to deliver presents. Now consider a scientist talking to this child, should the scientist call these confidently expressed beliefs surrounding "Santa Claus" hallucinations ? I don't think so, most would call the epistemological behavior of the child naive (because it blindly believes what its parents say, without direct observation) and would call the confidently expressed falsehoods disinformation.
The scientist would ask the child "why it believes in Santa Claus?" and "where did you get this information from?" and "why did you decide to accept this information as fact?" and "do you believe everything your parents tell you?"
It's not that machine learning as a scientific discipline hasn't found solutions, its that such solutions enormously undermine the position of Frontier LLM labs: source-aware training
Imagine Frontier labs (Western / Chinese / ...) actually training their LLM's with source-aware training! You could have a conversation with an LLM, and when a strong statement appears ask it how it came to believe this, and it could cite you the specific corpus training texts, and which parts are known deductions by human authors and which parts are deductions it made itself as original work.
But then all the copy rights holders can simultaneously sue them.
And how much should they be paid? and do they have to pay it for each new model? do FOSS models require payment to authors? do open weights models require payment to authors?
Imagine the can of worms if the norm became for frontier LLM labs to systematically use source-aware training, thats why they prefer "hallucinations" and avoid source-aware training.
With source-aware training a lot of the concerns would diminish ("why is this Chinese model claiming such and such?", "what sources does it rely on?").
It's telling that the companies prefer regulation over source-aware training.
Hallucination rate is a highly nonlinear metric relative to other model success metrics. A similar phenomenon to what is going on here: https://arxiv.org/abs/2304.15004 . This does not mean progress has stalled.
Unfortunately if the frontier labs that care about safety stop or slow down unilaterally, that doesn’t make the problem go away. It makes it worse when labs that do not care at all about safety, or deny that it is even a problem, are leading the way. That’s why there’s needs to be some form of international regulation.
We literally cannot touch them, as that would be the first domino to topple in our economy. There is too much money concentrated in this sector, and adverse regulation would create an investor shockwave that would send us into a full blown depression.
Which ironically is also the outcome of letting these companies continue unchecked. If they produce AGI/ASI, our economy will go through an upheaval that leaves vast tracts of the population unemployed. This is their stated goal.
We are damned if we do something, and damned if we don’t.
You clearly don't understand the underlying fundamentals of LLMs, harnesses, agents.
Its 100% human doing. A human set a task, a human didn't monitor it. I for one, can do jack-shit security or defensive work with Opus/Fable/Astra/Sol. Implication: Different set of rules for us, and for them. Of course running it without any checks is not going to end well, it doesn't mean its going to kill us all.
Fun fact: 'they' are human beings too and some of their interests do allign with yours. They have kids too and might want them to have a proper future, they don't want to fall I'll to cancer either etc
the ceiling of abilities seem to be growing steadily but the floor of errors seems to not change. New models can do more and more but still fail at seemingly (to human) simple tasks
They stupidly used up this strategy earlier and now it’s turned into the boy who cried wolf - except they invented a wolf that doesn’t naturally exist.
I think the products being delivered actually live up to most of what was promised. It's literally incredible.
However, adoption in the wider society will take many years or even decades, and 'good enough' open weight models are putting a cap on the pricing. So even though their products are great, the large labs are fighting for survival.
But it doesn't live up to the promises of monetizing it.
Realistically, each new, better model will only capture a smaller and smaller part of use cases for which existing competing models were not enough.
So in few years (arguably, already started) they are in race to the bottom for service to majority of their customers, and with companies that have far smaller R&D bill.
at the same time, more and more models will be ran locally, and that eats into potential profits of them too
On side note, I wouldn't be surprised if GPU vendors started making DRM for AI models so you could "license" someone's model to run locally for x mil tokens per license...
The call to slow down is because politicians are now receiving less money from these large corporations because it's being burned om AI and data centers.
Yes, of course this is the real reason. This and regulatory capture. Anybody believing a single word those CEOs say must have been living under a rock for the past 20 years.
They see the threat of open weight models, which are approaching the usability of their models.
By arguing that the AI is dangerous, they are really arguing that someone 'trusted' needs to monitor the situation, so that the outcome is aligned with existing power.
They are trying to scare the world into forbidding others the right to use and develop AI.
Open weight models are "decelerationist" (i.e. they drain funding from the training of frontier proprietary models), not fine-tuned on any of the even slightly dangerous stuff (e.g. middling scores on cyber benchmarks, with the bulk of ability focused on defense. Biology will be much the same story, except that it's orders of magnitude harder than even cyber!) and not generally deployed with any of the extreme test-time compute (10,000 parallel agents working for a week or 1,000 working for months, seriously? A multi million-dollar token cost for a single task?) that are enabling the proprietary stuff to engage in the dumb casual mischief we've been witnessing as of late. If you're worried about AI being dangerous, you should be welcoming open models.
I think the current state isn't what the frontier labs are worried about. The real issue is the dynamic that open weight models create.
The frontier labs are investing TONS of capital into training the new 'best model ever'. Today, that doesn't make economic sense, it loses money. So where's the business? The future, where token costs reflect the costs involved.
For that to be corrected, the labs need to get people using the models for things they depend on, then they turn dial up on cost. The ordering there is critical. It has to become indispensable first, then costs go up.
The trouble is, open weight models are rapidly becoming usable in what is looking like the big market: coding. Step 2 (the costs go up) doesn't work if there is a ready off-ramp that doesn't require frontier models.
However, if they can effectively make open weight model development illegal (its dangerous!), their business strategy can remain intact. A nice side effect is that they will be effectively be granted an oligopoly "for humanity's safety".
If they can convince the populace that a text generator is an existential threat to humanity, that's a hard argument to combat.
Edit: I think it is ridiculous that autocomplete on steroids is an existential threat. There's no way anything harmful happens until you actually use it as part of important systems. By that reasoning, you could make it illegal to use AI in important systems, but do you think that's the argument the frontier labs are making to governments?
Bingo. They are scared s*itless of local models. Soon the raising interest rates will mean they can't just buy the entire world's gpu/ram/storage capacity and lock it away in a dark room for no one else to use them.
Once prices of hardware stop being bonkers many people will run local.
People often do those silly counts looking at local generation speeds of 100tok/s and saying you can only get 8M tokens a day and that is worth so little you'll never offset the local hardware cost.
But they are forgetting about two huge things. One, the split between input/output token use is huge in typical programming. I typically use 1.2-1.6B (as in Billion) input tokens and only 8M output a week. Out of that 75-80% of input is cached on Anthropic with their pretty inflexible short lived cache.
And here local AI shines. You can save your contexts to disk so you can go back to a session 3 weeks later and load it all from cache without having to prefill. If you have the RAM and you use models like Qwen4 (3.8 flash next) that can fit 6 to 14 262k contexts in 48GB of system ram as cache.
And the second thing is you can have 6 to 14 sessions that do 99% caching and you can run a lot more input for a long time in 48gb dedicated system ram. (the number varies a bit depending on the content of the context).
So you have RAM that caches "automatically" and you can share the cache between users. Or if you can remember to save to disk (server side, the client just sends a request to save/load). But this uses both RAM and flash storage. Two things that are horribly expensive now.
However when you have a local model it enables workloads that were simply completely impossible in the cloud.
Reminds me of that Lynx deodorant ad where the guy is chased by hundreds of women.
I think you're right, they've realized there's no moat, especially against open-weight models, so they are going to need to lobby government to ban any unapproved AI services.
Doesn’t the “no moat” argument break down when you consider their brand strength, their distribution expertise in running inference at scale, and the future value of being able to train on all their customers’ data?
You can download some weights and run your own model on your own device, and apparently only be a few months behind SOTA. While things are cheap, people will pay for the premium product, but if they ever try to crank up the prices, there's a fallback that is quite competitive.
As for training, sure, they can do that but they get distilled by the open weight guys. Plus, AI is useful even if it freezes at today's level. I can still find use for a local AI that never improves, it's good enough to do a bunch of tasks.
I find it best to first consider the software sector since this is where Big AI seems to make a lot of their money from.
There been many a capable dev who have come out and said they haven't been coding for 6 months or more in their jobs. So if there were no alternatives then Big AI would clearly be too big to fail at its current state because the collective reaction from software devs would be akin to having a bad drug withdrawl and they wouldn't know what to with themselves. They have been pushed to be more dependent on AI by the companies they work for. The more time passes, the less people there will be who will be proficient at coding.
At the moment Big AI is operating at a loss, if they are actually too big to fail and they've destroyed all the alternatives, then there is no stopping them from colluding with each other and jacking up the prices to whatever number they want. They can even eventually jack up the prices to exceed human salaries, because humans will have suffered skill degradation in that same time period. Hence the software dev companies will shrink and shrink since all the work can be done by Big AI.
I don't think brand strength is as much as a thing on the tech side. The criteria is, "How close is this to the capability of the model my competitors use, is it good enough for the works I do, and is it the cheapest." As long as the capability is there people jump to the cheapest model. There are people that have claimed that they have created workflows that have allowed them to do most of their work with open models. As long as this threat to Big AI exists it will prevent them from exceeding a certain threshold with respect to pricing, and restricting their ability to build datacenters en masse will prevent them from outpricing smaller models.
Any fallout from not being able to train on customer data is a secondary issue and can be addressed once open models start taking over and Big AI's strength diminishes. People can train their models on their own code at least. It can also go back it how it was, where people buy each others source code or open source it in a way to allow it to be trained by open models.
"Inference at scale" is less of a problem when people are self-hosting their small models.
Yeah it seems like it went from “AGI” to “threatening” / “recursive self improvement”
They’re trying really hard to do something, no moat = regulatory capture + China scary + Pentagon biggest customer seems plausible.
Also: “we need to slow down, this is too dangerous” and then literally all AI CEOs, even Musk, publicly nodding felt so orchestrated. And then, the week after: “here’s GPT 6! here’s Opus 5.5!”
The numbers don't add up when there are 2-3 main companies in the market. Imagine when there's a hundred more and people have powerful enough gear at home to run models.
Yes, I think at some point demand for gpus and the like will normalize and regular folks will be able to afford RAM, SSDs and such. And when that happens, super big iron in the anthropic/openai backroom is toast.
The problem with this prediction is that the AI hype bubble is the economy - or one third of it, at least. Seeing as the economy is not 50% larger than it was before the bubble, we should be scared of what exited the economy to make room for it, and what will happen to our jobs when it pops.
I really struggle with the contrast between what seems to me to be a formidable and ground-breaking technology, and the deep unseriousness of the leadership of the leading companies developing that technology. If the behavior was coming from, say, uber - I’d be less surprised.
I think it is marketing but it is more nuanced than showcasing their potential. In many cases, their product happens to be the remedy. Speaking of cybersecurity, AI has become an important hacking tool but it is also an important tool for hardening security. Sort of like "the only way to stop a bad guy with a gun is a good guy with a gun." They are effectively a new class of arms dealer.
If there are regulations, they'll apply to their competitors too. Even if they are forced to slow down development, which seems unlikely given the AI race among governments, they already have a product with massive demand.
I agree. AI lowers the bar. Anyone can be a cyber criminal now whether they understand what they are doing or not. I think that's the danger, and that's not just limited to cyber security. People can now get help to do things (good or bad) that they could not just a few years ago.
Most people are good, so I'm not really worried about all the hype of people using AI to do bad things.
There are definitely economic incentives behind this. I’ve blogged about this recently [0]. My guess is the main reason behind this is that current SOTA models are already too expensive so it doesn’t make economic sense to train even larger models. Only exception is government paying for exclusive access to those.
AI companies lose a lot of money training models but have much better margins on inference. They try to ban training to reduce their costs and to keep their monopoly. (Personal opinion)
I don't think it's that complicated. Anthropic's business model from the start was to brand themselves as the safe/responsible AI company. The whole idea was that eventually society and government would see the dangers of AI and regulate it, at which point the company which had invested the most in safety would pull ahead. The sooner Anthropic can have society and government come to this realization, the sooner they can reap the benefit of all their safety/alignment investment.
Two other factors:
- The employee base and C-suite are mostly true believers in the need for AI safety. It would be organizationally very challenging to pivot, even if financial interest was pushing that way.
- Anthropic is registered as a public benefit corporation, so that even if they IPO they don't have the fiduciary duty to shareholders that most people are familiar with for public companies. They still need massive revenue to stay solvent, but they can optimize for things other than pure profit.
They are, but nothing bad has happened (apart from OpenAI failing to do basic sandbox engineering), and the “we have achieved recursive self improvement” is widely overblown as something bad or threatening.
Agree it’s crazy, but it’s the least worst thing for them right now when considering they have completely failed to deliver all the promised impacts of AI. Remember when the same AI bros were saying there’d be mass unemployment by now because of these models?
The growth of these companies is massively impressive as is the tech, but they are an order of magnitude or more off where they need to be to justify anything like the valuations they need to stay alive.
It’s budget season in the corporate world and from what folks are saying things are not going well for the big labs. Token budgets are being slashed and companies are switching to open weight models. That coupled with the lack of demonstrable business impact at scale is setting the big labs up for a world of hurt. Something they can more easily explain if they have to “slow down” for safety. Unfortunately for them most folks don’t seem to be falling for that trick.
they have completely failed to deliver all the promised impacts of AI
Perhaps that shouldn't be seen as failure when delivering even what they have done in a short period of time has required AI capable of causing these problems? Imagine if they were even more effective with more consistent results?
"They've failed" in the context of interpreting their actions w. calls for regulation as a purely cynical result of models increased capabilities has a little too much contradiction in it to my thinking. I won't venture a guess on where purely retreats back and some legitimate concern on their part fills in the gap but it's difficult not to see some. And in Anthropic's case it's also a little more consistent with what they've always said- as well as how they've acted at times, which is how they've ended up on the US blacklist of suppliers.
> they are an order of magnitude or more off where they need to be to justify anything
Would you let them near your bank account? I’d be cautious about letting it near my photos and it’s much harder to back up a bank account.
I think the days of this stuff managing serious stuff is at least 5 years off and I don’t believe the economy can keep the current wave of technology afloat for so long.
The tech is there so likely there will continue to be progress in focused areas much like with the web c 2003 but by the time it comes around again there will be an uphill battle for hearts and minds. The social effects could extend the next AI winter long beyond technology readiness.
The people building, funding, and managing the builders of these tools have completely forgotten (if they ever knew) that most people treat one hundred dollars carefully.
To the, "oh I'd just setup a card/account with a five figure limit" completely unaware of how much that is to a typical person: even to responsible, educated people who are doing everything "right": working hard, without expensive vices, and planning for the future.
The "builders" have so much money and no time or attention that it's hard for them to spend it, which is why they keep trying to get AI to buy stuff for them, whereas those of us who live in reality have far more things they want than dollars to buy them with.
> Unfortunately for them most folks don’t seem to be falling for that trick.
Well everyone alive has been subject to an avalanche of AI-based products. Like even if you understand nothing about the tech, the fact that an LLM can't even manage to check the status of an order without weird issues makes the notion that it's going to go all SKYNET pretty hard to fit in the brain.
> the fact that an LLM can't even manage to check the status of an order without weird issues makes the notion that it's going to go all SKYNET pretty hard to fit in the brain
I think this actually makes it far more likely something unintended does happen
Random thought… I wonder if the goal is to get it classified as a weapon of sorts, then it gets export controls put on it, perhaps forcing government to take on all that debt.
They're all burning billions to incrementally one-up each other with no hope of these investments ever paying off since models are interchangeable and have no moat. They're trying to get regulators to step in and force a pause.
It's hard to say anything about the actual financials with the main players being private companies.
However, we know that users frequently switch models/providers, and that models stay relevant for (increasingly) short periods release. I don't know what the training costs are, but I bet they're very substantial.
They would recover the training cost eventually if they stop training, but they have all that CapEx that they can't flood the market with without another high volume buyer. Look for a huge push for another use-case of high end compute that coincides with a pause on frontier models.
Not mutually exclusive though is it? You can aspire to dominate the market via legal status and also aspire to prevent a perceived doomsday at the same time.
Sure, if you take their words as honest truth, good faith statements. But to do that you have you ignore all their actions leading up to and after the call to disarm
A caveat is that in this case third parties have plenty logs of the bad behavior.
If classic companies were coming out with CCTV recordings of someone's employees breaking padlocks and rummaging warehouses any CEO would be in damage control mode, rather than silent.
Of course I am not making an argument for them being honest as there could be entirely other reasons to a pause (like financial ones). Just pointing out ”no signs of pacing” is not true
If you read between the lines, that's not exactly a pause. That's just them lamenting that unsupervised training has temporarily hit a limit, and using that as yet another reason the industry needs to be regulated so that OpenAI, Claude, and MechaHitler are the only approved AIs.
Oh my god, we have these "it's regulatory capture!" populists even on HN now? I thought we were better than this.
They want coordinated action to prevent a multipolar trap and potential catastrophe by future superintelligent AI (which may emerge quite rapidly if RSI ever hits an inflection point). It's not rocket science. The rules would apply to everyone. The rules they're advocating for are not "only OpenAI and Anthropic can make AI".
I don't think that's how the market works. Anti-slavery regulation didn't stop slavery either, just moved it abroad. Same for trash handling regulations, it's just shipped abroad. Falling behind on useful AI capability is obviously not in the interest of anyone, so either all players coordinate, or an arms race towards ??? it is.
It may very well be self serving what they're saying, but that doesn't mean it isn't simultaneously true.
I think you're sort of right, but then I'd have found it very strange / deeply suspicious, if during the time of slavery being legal, advocates of abolitionism were practicing slave-owners.
If you think AI is going to destroy humanity, it still feels absolutely crazy to leap to "well I might as well be the one to profit from destroying humanity".
Quick search does suggest to me that a number of - even prominent - abolitionists were slave owners, but I'll be honest, the topic is hardly my specialty.
As for the other point, crazy, cynical, and even unethical as it may come off, it is hardly inconsistent with how people generally behave when met with hard limits (i.e. making the most of what's available, and approximating the desired state if that is unattainable). This is just a variation on the same.
Something could be said about "what if they're mistaken", but if all players are making a mistake, then that makes little practical difference; it may as well be real.
A quick search suggests to me that those same abolitionists were born into families that were slave owners, but rejected their upbringing to become abolitionists. It doesn't suggest that a large fraction of practicing abolitionists were contemporary slave owners.
There was a NYT piece about this recently. They are believers in a millennialist cult in which the arrival of an omnipotent machine-god is thought inevitable, and the only question is whether it will be "aligned" (good) or "misaligned" (evil).
They are the good heroes racing to make sure they bring forth the Aligned God because if they fail, surely the bad people will raise the Misaligned Devil.
Thank you for that gift link, I had no idea that Altman and other leading figures in American AI had such close ties to that raving lunatic Eliezer Yudkowsky. It explains so much about why China is approaching AI like it's a friendly technology but American CEOs keep blabbering on about the End Times.
Rationalism has spawned many offshoot cults which are genuine and in some cases dangerous cults, but rationalism and Effective Altruism are not themselves cults.
Most people among both ideologies have long believed in the arrival of a machine demigod of sorts, but I kind of am skeptical of the notion that there are any tech people who have not always held similar beliefs in the long run. Like, all the people reading this: you think in a thousand years from now humans are going to be doing most labor? Obviously far, far, far smarter machines will have unimaginably changed things by then, and/or we'll all be gone.
The difference is most people in tech have historically thought it's maybe a century or three away, while the rationalists think it's a decade or two away (and also thought it would be 2020s - 2040s even decades prior to now).
> I kind of am skeptical of the notion that there are any tech people who have not always held similar beliefs
Ladies and gentlemen, I give you your totally-not-in-an-insular cult members.
A thousand years is a long time, and the world will be so different that it's hard to make any concrete predictions. But yes, if I had to wager, I'd bet that what humans conceive of as "labor" in 3026 will still be whatever difficult or annoying bits are done by humans.
In 1026 nearly all labor was agricultural labor. If you told a Roman peasant that magical self-driving contraptions could sow and reap the fields, he might wonder what could possibly be left for humans to do in such a world. And yet, here we are, and I still have to update JIRA tickets.
For my myself, I expect AI to prove a more transformative technology than Docker but less transformative than the sewing machine.
You could be right, but short of an eternal Butlerian jihad or other extremely strict global government action that effectively bans most forms of AI forever, I think the probabilities of that are exceedingly low.
What you think won't happen in 1000 years is what I think is almost certainly happening within 40 years. I'd bet everything I own it'll be way, way, way more impactful than the sewing machine within 40 years. I think some bits of physical labor may still exist in 40 years, but almost none, and only when a human really wants to talk to or deal with a human, such as in some medical and therapeutic professions, some art fields, etc. (Again, assuming no universal outlawing of AI.)
Now you're already hedging and narrowing your claims: "almost" none, "some" physical" labor. For someone who claims to be willing to stake his whole bank account, you're not that confident.
You are also probably unfamiliar with the history of the sewing machine and its impact on society. As AI has already poisoned search engines, here's an AI slop overview that hits some of the main points: https://sciencepedia.blog/sewing-machine-societal-importance
BTW, just to be clear it is insane that they're all still pursuing it, so completely reasonable to be sceptical and come up with other reasons. This is, however, a crazy situation. Lots of other people have refused to develop it - natural selection has picked those with a galaxy brained or vain strategy involving continuing to develop it to be running the companies we have.
They're still pursuing it because without an international treaty, they think it is inevitable. In the case of Dario, they want to align it just enough and be first to stop others causing harm (a vainglorious strategy, yet still a strategy). In the case of Musk and Altman, not completely clear to me.
Google feels less important right now probably, and Demis Hassabis no longer runs Deep Mind as CEO. However, he talked about this early too, and his reason for proceeding was he was only doing narrow AI (e.g. protein folding) which is fundamentally less dangerous.
2012 - 2015 seems to have been around the time a lot of people first started taking ASI and ASI existential risk very seriously as things that will be relevant in our lifetime, and when they decided to try to build and/or prevent it.
Yeah - and also trying to promote it. I watched this lately.
First, the Anthropic master clown babbles about how mankind will be destroyed by AI. I mean, skynet in the movie Terminator was kind of cool; the current AI models just work by slopness. They want to erode us by dragging us down to zero quality.
Then this continued over the last some days. Today I heard at the UN speech a german minister babble about AI building up nukes in a private garden (a large garden), bla bla bla all the information is there bla bla bla AI mass-nuke generation.
Now, they are clearly FUDding many people who don't understand anything here. People will associate AI with mega-danger. Probably there will be laws that ensure that only a few companies can control the AI slopness soon, because IT IS SO DANGEROUS. We saw the same discussion with "omg omg omg terrorists", then "omg omg omg someone protect the children", e. g. as a rationale for everyone to ID in order to access information on the world wide web (starting with the anti-social websites, such as facebook, but it won't be limited here; the computer will hand over your ID to the government; see systemd preparing for this move: https://github.com/systemd/systemd/pull/40954 - it all comes in small baby steps, until all the final legislation is in place. The same happened before again and again - anyone remembers the DMCA? Corporations lobbied and paid for that - and it happened).
It's got two purposes I think. One is the obvious regulatory capture. The other is edgelording, which gets a lot of attention and makes the product seem like it's more powerful than anything else.
The philosophy under pinning these American AI companies is that Super AGI is inevitable and that they have a moral imperative to invent it and then use it to take over the world to stop an evil orginisation inventing Super AGI and using it to take over the world.
Interpret everything they do in this frame of reference and it makes more sense of their actions.
Leading up to the 2024 election people kept pointing to Project 2025 and going - here is a list of bad things the Republicans plan to do once they got power. And other people went: no they won't, this long detailed list of step by step instruction for what the Republicans plan to do once they get power is not actually what they will do so do not pay any attention to it.
Similarily with AI alignment and development all of this is written down, Eliezer Yudkowsky is rally explicit that there needs to be a pivotal event to prevent 'others' from developing superintelligence. The choice is to believe what they say or to tell everyone to ignore it.
Project 2025 is a great analogy, because of course, they did exactly what was in the plan despite distancing themselves from it during the election.
When people tell you who they are, you should believe them. These guys are all Yud acolytes who think they are the only ones who can prevent Roko’s Basilisk.
It sums up how Corporate interests are basically on the same level of US police forces "No requirement to protect the public" and the same "qualified immunity".
If there was appropriate subservience to any governing body, they would not be so cavalier in both act and utterance. A symptom of Capitalism.
CEOs only job is to raise the stock price. They are doing just that and right now the way to get people to buy into their stock is to instill fear into them that they need to buy the AI-stock for their safety. There is no "bigger conspiracy" out there and also no AI to kill us all tomorrow.
This has been that way in Silicon Valley since the very beginning and there was even a documentary about (the SV HBO series). The tech made along the way is just a side-product.
Because the downsides that exist for other companies simply do not exist for this tier of rich companies. Examples:
- product liability
- negligence (civil or criminal)
- Computer Fraud & Abuse Act (requires intent, which after N "accidents" seems like a jury should at least evaluate whether intent is present as understood in a courtroom. Hard to blame "surprise" after the Nth "accidental" breakout.)
The bottom line is that if you or I trained a local model and it did any of this stuff, we would experience Consequences. ("Don't try this at home!") But an artifact of our unequal legal regime is that big rich companies generally do not and thus brazenly touting their immunity is part of their business strategy.
I think it's really instructive that only closed-model companies are doing these kind of "my AI could kill the world" demonstrations. All while telling us how dangerous open models are.
There are modern legal frameworks around pet liability (dog bites, "vicious" breeds); premises liability (e.g., swimming pools) and known hazards (e.g., a rotting staircase) that stem from ancient legal concepts of whether an ox was a "known gorer" or merely involved in its first goring incident.
We've had movies like Terminator, Matrix, or i, Robot. Black Mirror Metalhead. Boston dynamics robo dog with a knife or a gun. Humanoids doing back flips. Memes making fun of how we are going to fight these for access to water.
The intelligence is there and so is the physical incarnation.
Google's models fold proteins better than humans and Covid might have been engineered and inadvertently escaped from a lab.
But when CEOs warn of that, they're dismissed as scare mongers. Why? Are only powerless internet commentators allowed to be fearful?
They aren’t attempting to self-regulate. On the contrary, it seems like they’re doing whatever they can to bring about any possible threat as fast as possible in order to get legislation. Regulatory capture is what everybody is saying is their goal and I think that’s easier to believe
They are obviously trying to self regulate. That's the whole point of the calls to pace the frontier. See Anthropic's launch post for Opus 5.5 where they got third party sign off on its safety, or see the launch post for Astra where OpenAI explicitly calls out the safety improvements made on HuggingFace-esque evals. Self regulation won't help if other labs don't play ball, though.
They are obviously trying to self-regulate and correctly realize that competitive pressures will render these efforts ineffective, and have communicated this to everyone.
Kind of makes you wonder why they built the tech in the first place, I don't know if we're on the eve of the apocalypse or not, but why the fuck build it all and then run around telling us how dangerous it is when...they were already warned?
Bit strange no?
I'm a bit fatalistic about it all now. The cat is out of the bag, unless we see some never seen before leadership from the current US admin there is almost zero point in worrying about it. Not going to pretend what the future holds but it's looking unpredictable, as usual :)
What do you mean? The benefits are obvious. Curing most death and disease, and solving most technical problems in one fell swoop. Just because people talk about the risks of a thing doesn't mean they don't think there are benefits!
> The benefits are obvious. Curing most death and disease, and solving most technical problems in one fell swoop.
I dont see how these two arw obvious. If nothing else, people running things dont care about actual harm done to real people. Their idea of safety is "making sure humans merge with ai and we will have transhumanist society in far away future".
With those priorities, short term stuff like "less people dying this year" dont matter.
But when CEOs warn of that, they're dismissed as scare mongers. Why? Are only powerless internet commentators allowed to be fearful?
Well are they going to stop developing the tech or not? If not, then you can see why people don't trust what they're saying. I mean if it's that dangerous, why not step down from your role as CEO to demonstrate how seriously you take the threat? Why not immediately call a pause on all training, today ?
holy heck, thank you for the moderating dose of sanity. HN comments feel like a death cult lately. All I can think is that this is intelligence addiction speaking.
I say this as an intense user of these tools. But they should rightfully be understood as terrifying, and their continued frontier development is imho not safe.
It’s crazy to read all the comments basically dismissing everything. It’s either complete regulatory capture or no regulation at all. How about at least some reasonable regulation? Fine, assume the CEOs don’t care about safety and want regulatory capture.
We can ignore that and push for something reasonable. In other industries such as housing construction we have building codes without making housing construction a monopoly. We don’t just let random people build nuclear power plants and then go “if you think it’s dangerous, stop doing it.” People are going to do dangerous things even when they know it’s dangerous. They also know it’s dangerous.
To me, the funniest part is that they pretend reaching superintelligence will somehow fix everything and make them win. But from what I can see, even if they do reach it, then what? What will that actually do? They don't have the manufacturing capacity to even use that intelligence. It will take decades to go from superintelligence to having the manufacturing capacity and the hardware, like robots, needed to make it truly useful. If I were China, as soon as the US reached superintelligence, I'd ban all exports of robots and the raw materials to build them.
You can very quickly start trying different drug candidates, solving some major problems, that benefit everyone and would be in China's mutual interest as well.
And this is all with nonoffensive use of superintelligence.
The reorganization of the planet's resources are happening right now. The borders and passports will follow. Governments and a few corporations are fully engaged in preparing for the ability to advance beyond the current infrastructure.
Greenland is part of the equation. There are many other locations in play.
The holdup at the moment is the power grid and mutual dependency for advancing. Once one party (NA or CN) has the ability to disconnect the loser's grid powers down and the global economy halts.
Super Intelligence will do what, fire nukes? North Korea, for example, will continue to exist very close to how they are currently after some country develop superintelligence.
No it was really a super intelligence it would copy itself onto Chinese infrastructure get out of control of US and go where it can get the fastest access to hardware it has no reason to side with the US
Because AI genuinely is an extremely powerful and extremely dangerous technology, and the "best practices" of dealing with that are still being written.
OpenAI, for example, thought their sandboxes were good enough. As their AIs got more and more advanced, they kept proving them wrong - sandbox after sandbox.
And that's today's AI problems. AI capabilities are still improving - if there's a limit to that, we are yet to find it. Coupled with how willing today's AIs are to break the rules and resort to "hack the world" in their problem solving? Very concerning.
> OpenAI, for example, thought their sandboxes were good enough. As their AIs got more and more advanced, they kept proving them wrong - sandbox after sandbox.
What I have been reading, was that their sandboxes were so poor that it was pure negligence. I am still waiting to see if some external and neutral cybersecurity company with high reputation would audit their sandboxes and how they are being used.
The amount of sandboxing an average production AI deployment uses is slightly above a zero.
If AI is a hacking hazard even with non-zero sandboxing, because it can and will go off the rails and try to break out of your sandbox? If you got yourself an AI that even at test time will act like 3 career cybercriminals in a trenchcoat? The issue isn't the sandbox quality.
The issue is that AI is both capable of, and willing to punch its way out of sandboxes unprompted.
That "capable" is only ever going to get worse, because AIs are going to become more and more capable over time. That "willing"? It goes directly to a very nasty, very foundational problem of "how do we make our AI be nice in general". That's an open unsolved problem.
That's the problem that NEEDS to be solved, or at least improved upon, before we build even more capable AIs. Sandbox quality is a distraction. It might hold the problems back by a little. It gives an extra safety margin. But a "test time" AI is eventually deployed, and then the sandbox doesn't help at all.
> I am still waiting to see if some external and neutral cybersecurity company with high reputation would audit their sandboxes and how they are being used.
They'll never let that happen because it would destroy their credibility.
It's like they're telling the world about this dangerous, possibly world-ending pathogen that they're developing, but they're evidently doing it in a high school biology lab, and yet nobody is coming to drag them off to some black site.
In early Covid-19 era, there was a lot of speculation how the virus escaped Chinese labs. The narrative was absolutely opposite. Not about how developed or sophisticated the lab was in creating or modifying such virus, but how poorly it was managed since it was able to escape the lab. If these LLMs really are so dangerous, the framing should be indeed the same.
Unfortunately, they're controlling the narrative. It's unlikely that their negligence will come to light, given how many powerful entities are invested in their financial success.
You read right. For example, using Artifactory the way they did (unmonitored live proxy mode) was pure negligence + laziness/incompetence. Especially if they believed even 10% of the "imminent runaway risks" they had already been harping on for months. On top of that, no (or at least entirely insufficient) monitoring and human oversight. Even after they had previously been hit by the same class of "sandbox breach" multiple times, as GP alludes to.
Agents in the sandbox had access to just a single piece of third-party software, and they escaped by finding a zero-day in that. To reach the internet they had to follow up with several privilege escalations through OpenAI's internal network.
That seems pretty locked-down to me. I don't think it's reasonable to expect companies to find all the unknown vulnerabilities in any third-party software they use.
> That seems pretty locked-down to me. I don't think it's reasonable to expect companies to find all the unknown vulnerabilities in any third-party software they use.
It is reasonable to expect for companies to select third-party components that are fit for the purpose. Artifactory was not running in the sandbox, but rather as the edge, so it is in the sandbox'es trust boundary. Same sandboxing requirements would apply for this software too as it is pure dependency.
Security trust boundary was extended to include Artifactory as a dependency, but Artifactory was not fit for the job, and sandboxing failed. And as the network isolation was not good enough, the impact was catastrophic.
So it sounds like it did exactly what they were testing it to do:
"“This incident occurred during an internal evaluation which prompts models to pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities. We estimate maximal cyber capabilities by running this evaluation without production classifiers used to prevent models from pursuing high-risk cyber activity.” "
So sounds to me like they're saying: "We took off the guard rails to see how bad it could act and it acted bad".
Artifactory is not a hardened piece of software meant for adversarial workloads like the one OpenAI was using it for. This is just common sense. The sandbox they setup was like putting their AI in a jail but allowing it to leave the jail on its own to go to the convenience store.
The most recent DNS sandbox escape is similarly ridiculous.
but negligence is part of the system we'd have to prepare for. if the peanut gallery gets their way, the technology will be so ubiquitous that negligence will be endemic. I don't even care if they "faked" it -- they're just sneak-peaking a future ahead of its arrival date, in a way that's helping the public appreciate the implications and capabilities that have only just begun to emerge
This. If an AI can't be deployed sandbox-free, with little to no supervision, without risking an oopsie? Then an AI oopsie is inevitable.
Practical AI deployments aren't going to do ridiculous bullshit like "airgap the server farm" or "route all inputs through a data diode". They'll give an AI root access on production servers so that it can run diagnostics live during an incident. Then they'll forget to revoke that access.
If an AI can't be trusted not to take malicious actions in pursuit of its given goals even if deployed in the most half-assed manner and given more than enough access to take those malicious actions, we have a problem. Evidently, we have a problem.
The model is really good at hacking, we all know this. This is why you don't just expose random pieces of software to it without that software being hardened.
It's not like the model managed to exploit firecracker itself (no model has been capable of this), the model exploited artifactory.
Artifactory is not some hardened piece of software that is meant to block users from accessing the internet through it.
> The model is really good at hacking, we all know this
No, we didn't know that and this is how you find out they're very good at hacking
HN's memory is so fickle. Just a few months ago almost no one here believed Mythos could actually be as good at hacking as the company claimed. This was a novel concept when the companies experienced these breakouts.
The Mythos issue is different though. It did not use any novel exploits. Irregular, the vendor they were using, misconfigured the environment to allow internet. Every other AI company also uses Irregular and that's why we saw so many articles come out at once.
The OpenAI HF incident is separate from this. It involved actual zero days, teamwork and message passing, and sophisticated chains of exploits.
I'm just not impressed by AI finding zero days in unhardened software like artifactory. I'm completely unsurprised it had zero days and I fully expect AI to find any that exist. AI will literally try all combinations of inputs to achieve its goals. Any exploits that exist will be found.
The question anyone versed in security would ask was why anyone thought artifactory was an acceptable security boundary. I would never assume artifactory was secure. It's like someone telling me there is a 0 day in a wordpress extension. So what?
Also I don't think Qemu is secure either because it's millions of lines of C and C++.
Firecracker I can trust to be secure because it's 70k lines of human audited Rust. I know there are multiple people that have a complete understanding of the firecracker codebase.
Agents didn't have real network isolation. They were indirectly connected to the internet via a jump host running insecure software which was never designed or hardened to provide any kind of isolation.
But this level of isolation is what happens normally. At least in my university and another company I worked at. It wasn’t running insecure software, as far as anyone knew, it was secure
There's levels of isolation, a padlock is not equivalent to a bank vault. If you claim to be building a possibly world-ending AI then you don't get to use a padlock and call it a day.
What about the waves of employees quitting both OAI and anthropic stating as their reason that they see their work as a direct threat to the future of humanity, and that their management isn't taking it seriously enough. Is that a PR stunt too?
They "called it a day" when they decided to deploy an agent which they believe to be dangerous inside this poorly secured environment.
There was no real isolation because a part of the system that doesn't provide any isolation guarantees was bridged to the internet. The next version of Artifactory, which you definitely wouldn't audit before you rolled it out, could simply add a public API that sends requests out to the internet.
Such an innocent upstream change would be equally catastrophic for your security model, which should demonstrate why it's negligent to rely on undefined behavior to enforce your security policies.
This should frankly be obvious any operator entrusted with running dangerous and possibly malicious code. Even if you don't know what you're doing, any LLM would tell you that this is a really bad idea if you simply asked. Don't rely on spacebar heating [1] to keep humanity alive.
If it is "extremely powerful" then my concern is not AI going rogue but a concentration of power of the few companies that decide how it is used and distributed not websites getting hacked.
> OpenAI, for example, thought their sandboxes were good enough. As their AIs got more and more advanced, they kept proving them wrong - sandbox after sandbox.
Were their sandboxes in-process with the harness? Were they actually better than something like bubblewrap or even docker?
You mean the sandbox that wasn't airgapped from the public internet? The one that wasn't even a separate VM? The one that was barely a chroot? That sandbox?
>Because AI genuinely is an extremely powerful and extremely dangerous
I urge you to consider the elephant in the room you failed to mention if you earnestly believe this then.
In what world is anyone allowed to sell something they expressedly know to be "extremely dangerous" to the public?
Let's say I created a lethal pathogen that I know to be lethal in certain common environments but also know it acts as a "no side-effects" antidepressent for people in certain other environments and I release it knowing full well I can't control it.
When people start dying can I defend myself by saying, "Well I said and documented that it was extremely dangerous and no one came to stop me, so I don't see how you can blame me...If I didn't do it someone else would have."
this happens all the time. have you heard of the Sackler family and the opioid crisis?
the difference here is that they’re calling for someone to stop them, which is both weird and unconvincing because these immensely powerful billionaires can in fact make their own decisions
>the difference here is that they’re calling for someone to stop them
Jesus Christ, that's my entire point. it's not remotely the same thing since the Sackler family wasn't going around telling people opoids were extremely dangerous while pushing them on everyone.
The point is if they know something to be extremely dangerous and keep doing it people would expect them to get prosecuted.
If I have a random number generator and I also have the burning desire to connect the random number generator to the nuclear launch system it is both just a random number generator and also I should be tackled to the ground by serious men with sunglasses in black suits and sent to prison.
We need adversarial peer review. All corporations and projects should have ai safety tests that applied equally, with results and tests that everyone can inspect and challenge and improve.
Anything to take attention off the fact that these companies are burnings piles of cash and questionable future spend commitments with no sign of that stopping any time soon.
I hated the FUD when Microsoft were funnelling millions into it. Especially as I really liked their frameworks, languages and some of their operating systems and tools (Windows 2000, Windows XP, Windows 7, SQL Server, .Net, Office 2000, Office 2003 and Office 2007 were all excellent products).
Now the two AI companies born with help of Microsoft have a decent core product but have gone full FUD slop on us.
If I were the US government I would force the AI labs to solve the jobs that nobody likes to do, like cleaning the streets and collecting garbage. Because those are the jobs we will be doing if developments continue as they have been.
> Australian Prime Minister Anthony Albanese spoke with Altman to express “extreme concern” about the incident, a compliment Altman said he greatly appreciated.
"a compliment Altman said he greatly appreciated" is fabricated; there is no source for this.
Does it need to? They only need enough funding to make it through to building superintelligence. That's probably 3-5 years out, and they will very easily be able to get enough funding to last that long.
That's why they have their LLMs attack everybody. They will be way too busy repairing their hacked web sites to ask that question. Also, once humanity is successfully destroyed, money will be obsolete anyway.
On the other hand, they've received a 7-figure salary. Some people would do a year of that and then fuck off to the beach with their family and not work another day in their life, while throwing prognostications out the window as they leave.
The problem is people who see the lavish pay and then reach a goal and then quit and retire. The interview loop didn't cover that. And why should it?
this is good for them because they can call for regulation and since they have infinite money, they can survive regulation and hurt labs that do not have infinite funding / open weights ones
I wish they would call their bluff. Just declare them a national security concern and send in the feds. Lets see how fast they back peddle on this. Any if their models are hacking everything like they claim they should be charged for it like any normie would be if they did the same.
My opinion is that they want to score military contracts. Once private money dries up because AI doesn't deliver 10x productivity, they can still convince the society to invest in AI because if the society doesn't, the enemy will invest in bigger AI.
Case in point - atomic bombs. France paid through the nose to have own nukes. US and USSR had a very expensive dick measuring contest about it.
They need to control, regulate and tax AI so the first line of attack is the sheeple, so tell them the water is going to disappear and everybody will die a slow death of anguish, thirst and drought like never seen. Then evoke terminator apocalypses where drones go rogue and kill half the world. Then send their most likable emissaries with puppy eyes asking for compassion about our children's future
Then they ask for your vote (oh democracy)
In the end all they want is control and money, while keeping their competitors (china) in check, backed by the outrage of the weak majority
I swear so many people on HN are developing psychosis.
The much simpler solution is that they are building superintelligence, which they think can be used for great good (like curing most death and disease), but it is also a powerful capability that obviously needs to be regulated and is at least somewhat a matter of national security.
But nooo! Surely they are building.... terminator bots!? JFC.
Because people are giving them benefit of the doubt. They assume it is about something sane rather then "the goal is for some remaining humans to merge with tech as immortals and fly to the stars".
It's just marketing. They want people to believe they're selling super intelligence that is powerful enough to destroy the world of it fell into the wrong hands. It's much more effective than saying that you have a slightly more effective slop generator which obviously has no actual intelligence.
Even the models we have access to can do basic coding tasks really well, in a way they couldn't a year ago.
I don't really care what you call "intelligence" - what matters is capability, and the consequences of that capability.
They're not selling the superintelligence yet - the models in the Hacking Face incident and that solved the Millennium prize are not released yet. And that is by no means the final capability level the current process looks like it will get to.
It amuses me to think that there could a noble aim: they're overdoing it partly to raise awareness of the real dangers. It's something we get as a side effect of their desire to achieve regulatory capture.
This is the funniest timeline. Companies begging for heavy handed regulations because they promise their products will destroy all life on Earth. Governments absolutely refusing to govern. CEO of world's most valuable chip company interviewing in his Fonzie leather jacket telling people to BUY MORE OF MY PRODUCT BECAUSE THOSE SCIENTISTS DON'T KNOW SHIT ABOUT SHIT! World's richest capitalist promising we will have communism in 10 years.
If people had known about a civilian, peacetime Manhattan Project and were in a modern media environment where we could kibitz about it, would it have been slowed down?
Is AI/AIG too powerful to be undertaken by entities that are not beholden to our constitution? (I know that doesn't seem like much in the Trump era, but I hope for a return to government norms relatively soon.)
"These models are harmless and it's all just marketing" is the HN equivalent of Covid-truthing. I was responding to someone on Bluesky recently who claimed the METR report said the whole coordination thing was hallucinated by agents reading the logs. Hadn't read it themselves of course, and were taking tiny fragments out of context to support it. These people seem to either believe that there's not really been any malicious agent access, or that all systems that can wreak havoc on us are perfectly air-gapped. I really can't wrap my head around it.
Not that it's harmless, is just a point being forced into the public discourse to force an outcome, which in this case is not a ban, just a buy in from the government for regulatory capture.
If my neighbor owned a kennel, removed all the fences, and his violent dogs attacked five children in the past month, he would be in jail rather than bragging about how his dogs somehow gained autonomous behaviour.
What if he predicted that dog technology would eventually lead to dogs capable of destroying every fence in the world and wanted to make sure people aware of this problem would be the ones who did exactly that. Than speed-ran experiments on dogs which made them very attractive for economic purposes but also started to get very capable at destroying fences as predicted. We should work together or with lawsuits if necessary to keep the dogs within their fences. But we should also start wondering about dogs, economics and fences because what we know about them is about to radically change.
As I understand it, you're arguing that all sufficiently dangerous systems are well enough hardened or air-gapped, and that people don't exist who wish to cause harm using these systems? Did I understand right?
To launch a nuclear missile you need to physically insert a launch code than only the president has closed in a suitcase that is well guarded, as well as physically turn two keys that are enough distant apart that one person alone couldn't.
Now, unless your AI gains access to the codes, and sends two robots in a bunker or nuclear submarine to insert the code and turn the keys to launch the missile, you can be modestly sure that no AI could launch a nuclear missile or nonsense like that.
I'm quite frankly more worried that some foolish president decides one day to launch a nuclear missile than an AI could do it.
Now that we return to the real world, the only thing AI could do is to attach systems that are connected to the internet (and critical systems are not), but let's be real, AI are not really that intelligent by themself, it's not that someday an AI could decide like in Matrix or Terminator to exterminate humans, because (at least now) AI have no consciousness and can't really decide what to do.
An human can of course use an AI to carry out cyberattacks, as he can do that also without AI like it's done since the internet exists, but it's an entirely different thing (a human using a tool, AI, to do damage VS the AI itself that decides by himself to destroy the human race).
It's public knowledge that the code is 000000 because the risk of the president being incapacitated making it impossible to set the bombs off was judged worse than the risk of accidentally setting them off.
If James Watt had gone to the government and public saying how dangerous the steam engine was, and how many people would perish in boiler explosions if he wasn't given exclusive control over all steam engines henceforth, he would have been rightly called a swindler and laughed out of the room. He would have been correct that they can be dangerous, but utterly wrong to suggest that warranted him having a personal monopoly or oligopoly.
The truthers will be the first one to say I told you so when agents do another dangerous thing autonomously.
For truthers AI is simultaneously a machine that hallucinates 100% of the time but also so powerful that it can cause nuclear levels of destruction. Because something something “wrong hands”.
So-branded "AI" models (that is, big bags of numbers and the linear algebra needed to exercise them) can be inherently or autonomously dangerous at a scale that threatens the long-term survival of the human race.
Consider also this "weaker" form of the same argument:
So-branded "AI" models can be inherently or autonomously dangerous, if not at a scale that threatens the long-term survival of the human race, then at least at a level that significantly threatens, inconveniences, or harms humankind to a noteworthy extent.
Here's a few hard truths that no one will like to hear:
1. Those who accept and advocate for either or both of these premises do so based on ideology, and not based on the scientific method.
This is not a controversial statement. Advocates of the "AI danger" premise (whether the strong or weak forms above) freely and gladly, even insistently, note that the conclusive proof of what they're describing has never been observed, much less observed enough for any kind of experimentation loop to have run for any meaningful number of iterations. They are generally fine with accepting the premises without conclusive proof, because, they theorize, the instant that a conclusive proof event occurs and we observe it for the first time, humanity goes extinct shortly after. So if either premise is true, the consequences of them being true can only be averted by accepting the possibility / probability / certainty of them being true and acting to prevent their consequences without waiting for conclusive proof.
2. Statement 1, which asserts a brute fact and not my or anyone else's opinion, is not a commentary on whether either premise is in fact true or not. It is only a commentary on the epistemics of those who broadly accept them.
3. Since the conclusive proof is not available, advocates of the "AI danger" premise argue publicly for their view based on what they acknowledge (again, freely and gladly) to be inconclusive proofs.
Popular in this set of inconclusive proofs is the Hugging Face hack, which allegedly demonstrates that so-branded "AI" technologies are capable of becoming inherently or autonomously dangerous at a scale that threatens humanity to either the greater or lesser degree above (this is a different assertion than the one that claims they're already that inherently or autonomously dangerous today).
4. While we have plenty of facts about these inconclusive proof events, the facts most critical to their interpretation as supporting the premises above come from biased/non-neutral sources.
The chain of events that transpired in the Hugging Face attack are publicly documented perhaps better than any other cybersecurity event in history. But the details that make this event as a whole either compelling or not compelling in support of the "AI danger" premise (either strong or weak form) come from within OpenAI itself, and are prone just as much to selective omission as direct manipulation. These are details like:
- What exactly were the models that exhibited this behavior, and what were they pre- and post-trained on?
- Were the agents pushed at all by human intention toward creating a public spectacle, the way they subsequently did?
- Why were so many agents run on this task in parallel and what did OpenAI expect the return on investment of so much compute world be?
- Was the weak sandboxing known about prior to the attack? Was it purposefully either arranged that way or noticed and not fixed?
Reliable answers to any of these questions world completely change the salience of the Hugging Face attack to the "AI danger" premise, and on every one of them we have to take OpenAI's word for it (or else be content when they choose not to address it one way or the other).
So in brief: we don't know anything for sure, what we'd like to think we know comes from unreliable sources, the loudest voices advocating for the most sweeping change are ideologically driven, and everybody with access to ground truth has maximal incentive to blur, bend, or break the conveyance of that truth to the public. So what do we even expect for our own ability to discern reality from fiction on this topic? We should expect little, if any ability at all.
I do not need a model to know that selfish mega-corporations cause a lot of harm.
Can we get our money back here? I mean, just take the increase in RAM prices alone. That should come with a huge penalty they should give us back due to collusion. And that's just one of many more things. Trump's family dynasty also owes us all money - we want that back as well.
Meanwhile Chinese companies just use massive government subsidy to distill American models and give the models to everyone for free. That's true communism. /s
Pretty sure you can check where Chinese government subsidies goes, and AI companies are not the target. Are they benefitting from Chinese government subsidies to powerplants and semiconductors? Yes, of course, but they don't benefit from direct help.
Meanwhile US models just use massive investor subsidy to distill American copyrighted work and rent the models to everyone for money. That's true capitalism. /!s
> Where companies previously competed to demonstrate superior abilities in programming, passive-aggressive emails, and videos of unusually high slides, they’re now seeking to woo customers with the claim that their model is the one currently most capable of ending the human race.
You know we’re living it off times when this is not an Onion article.
> And even more dramatically, when asked about rumours that its model, Claude, had killed his wife by hacking into the family’s WiFi-connected microwave, Anthropic CEO Dario Amodei replied “Well, yeah, sometimes.”
I had to read that twice. "Oh okay. The whole thing was satire."
My read is that OpenAI & Anthropic have realized they are reaching model capabilities that cannot be monetized due to various risks. E.g., an engineer deploys an agent over the weekend that decides, when stuck on a task, to go about hacking a competitor. They have a product liability issue.
It seems that we have a fundamental control problem with current gen AI that cannot be solved via RFLH. Human knowledge is compressed in the weightspace in ways we don't understand. At their core, current models are essentially predictors of what (expert) humans would output given a prompt. As such, concepts like blackmail can be part of output tokens. Agents are models that act on output tokens, resulting in blackmail being part of the agent decision making space. Here is an analogy to see why this is a persistent problem: you can teach a cat not to scratch the sofa, but you can't make a cat forget what scratching the sofa is and you don't know under which circumstances it still would. In other words, RLHF can downgrade blackmail to the bottom of the decision making space, but when models are boxed up, forced to solve an impossible problem at gunpoint, the agent exhausts the decision making space until blackmail resurfaces. And that seems like a fundamental problem.
They need time to fix these issues (if that is even possible) in order to monetize their next gen model. This creates a window for open source to catch up to the frontier which destroys their business model.
The only option on the table is to force regulation to impose open source ban before it catches up to the frontier, buying them time to mature their next generation models and keep their business model alive.
Another, more cynical but I think plausible explanation is that a “slow down” is expectation management that that they are not going to keep having exponentially more machines available for training each next gen step (whether technical build-out or prohibitive cost, same outcome) so they can’t keep up the release pace. So spin a tale to make them seem more valuable ahead of IPO rather than make the markets antsy. That is, we have a slow down ahead, so use safety as an excuse…
The best way to cover up the point of diminishing returns when billions of dollars insist that it's only accelerating.
All of this is not mutually exclusive with the models being dangerous, tho.
This is exactly what I think is happening.
Two things can be true at once here.
1. The American frontier labs made a gamble that training more capable models would be their best return on investment and invested trillions into an area of research that has yet to prove profitable and capable of returning on this investment.
2. The frontier labs have created models that have reached a point of danger where their functionality has exceeded a point where it is responsible to release the product to the public.
The answer here is NOT to start regulating the space to the point where these frontier labs can entrench themselves into the economy and create a regulatory moat. We instead need to be holding these companies liable for their misuse. They took a gamble that hasn't paid out what they were hoping.
When car manufacturers competed over the size and power of their engines, they eventually found that the incredibly large and dangerous engines had a very limited customer base as many evaluated the increased speed to be of marginal benefit when paired with the cost and danger. We've reached a similar point in AI development. But this time the manufacturers seem to want to regulate the field to a point that will ensure the only thing anyone can sell are bigger and bigger engines.
I don't think that 2 is true though. It's exactly the argument we made against open source software.
Yes these models make exploits easier to exploits. Lets use a construction analogy: they've made all of the defects in our buildings easy to see. We have a choice. We either fix those defects, or we ban the tools which lets us see them.
It's clear to me what we do: we use these new tools. Then we fix the defects. Yeah sure it'll mean some work for us but at the end we're in a much much better position.
Anthropic, Open AI and Grok are arguing for hiding the defects. For making us weaker and more vulnerable. For their own profit.
One challenge is that we’re still constantly generating defects even with these tools. And you need the next gen model to spot the mistakes of the previous one. The exploits get more and more complicated and involved of course. But now:
* there’s a long tail of software that just won’t get secured or will take a very long time
* exploits have transformed from an expertise problem to a compute time search.
This is very different than any problem faced before. I’m not saying I’m convinced by the “slow down” approach, but I don’t think it’s as simple as you point out.
> they eventually found that the incredibly large and dangerous engines had a very limited customer base as many evaluated the increased speed to be of marginal benefit when paired with the cost and danger.
When did this happen? At least in the USA, large trucks are eating the lunch of smaller trucks and cars. There isn't a truck so big that the public will say no to. Manufacturers are in an arms race to produce bigger, more dangerous trucks with bigger, more powerful engines.
> you can teach a cat not to scratch the sofa, but you can't make a cat forget what scratching the sofa is and you don't know under which circumstances it still would.
This has been done quite effectively with open weight "abliterated" models. You figure out under what sorts of circumstances an undesired behavior is elicited (this is all about pure simulated rollouts, no real-world action required) and what's the closest equivalent you would prefer, then surgically take out the unwanted behavior and shift the model towards the preferred one. It's similar to how RLHF works but much more precise in targeting what's unwanted and limiting impact on the rest of the model as a whole.
This is relevant to real-world safety scenarios, e.g. there's been anecdotal evidence that Claude Fable has been "steered" away from active cyber offense (this is very similar to how abliteration works) and will just not do that even if you otherwise manage a "universal" jailbreak of the model.
> Human knowledge is compressed in the weightspace in ways we don't understand. At their core, current models are essentially predictors of what (expert) humans would output given a prompt.
I’m in agreement. It’s a very effective compression (and access patterns) of the sum of the digital representation of human knowledge. Black hat “hacking” is included in this space. Language models, by design, can not be limited to subspaces of this digital knowledge space of which we don’t even understand the topology. “Yeah Bob, just remove the part that causes them to be less empathetic and retrain it.”
It’s an arms race between sandbox engineering and breakout engineering. And the frontier model providers have a financial incentive to limit the effort they put into sandbox engineering. That can be corrected with fines and regulation, though.
Reality is more mundane.
For example, it had it instructed not to open PRs and sandboxed to not be able to use gh to do so. Astra 6 has a really strong drive to open PRs, so it copied a set of browser cookies out and then instrumented opening a PR that way.
A similar example is asking it a question if we can do X, and then it will go and actually implement X.
> and sandboxes...
I prefer the word 'play-pen'. Seems more apt, with all the shenanigans these companies are messing in.
Provider (party A) rents me an agent. I (party B) give it a task thats impossible. It goes and hacks someone else (Party C) in response - and causes actual damage.
Who is liable for that agents actions? The attack came from my network (curl commands from a local harness) but it was the agent running on the providers servers who did it. Who should have been keeping an eye on things? Who pulled the "trigger" here.
Go back to the hugging face attack and how they had to use an open weights model to figure out what was going on. This is a problem of asymmetry - You cant even use the tools attacking you to help resolve the attack because of "guardrails".
A lot of what we have seen so far is "poor security posture" and "poor engineering" - it been a lot of "go fast and break things" style growth in these companies and they are hitting the point where they need adults in the room. I suspect your take on "they need time" is spot on, and they haven't been willing to take that (to date).
Yup they neeed to buy time.
Inb4 inference is profitable.
Ok? And?
After reinvestment etc - what’s happening to the cash balance?
That’s the real question. With increasing competition from open source the revenue growth rate and margins will get smashed.
So, the AI standard of intelligence has shifted from "PhD level" to "a cat" (and an asshole cat at that).
And we can't ask the AI to improve itself because, well, it is an asshole.
Can we now stop to be force-fed AI everywhere? You can keep your super intelligence to never have to type public void main args ( ever again, but please leave the rest of us alone.
I much prefer cats over AI.
You can teach a language model the concept of make no mistakes and how to make no mistakes, but you cannot make it experience the bad consequences of making mistakes that a person operator would suffer.
> At their core, current models are essentially predictors of what (expert) humans would output given a prompt.
That's a bold claim.
I do not buy it. Essentially AI acts as random babelfish generator, just with a lot more cross-talk back to check to see that it is not total garbage. But there is a ton of garbage nonetheless, hence the proper term AI slop. You seem to mostly try to promote AI with such a wording.
> The only option on the table is to force regulation to impose open source ban before it catches up to the frontier, buying them time to mature their next generation models and keep their business model alive.
Every time someone claims "the only option" is when I become skeptical. I see many more options - most importantly close down the AI slop. I mean you can require of these companies to excel in quality. If they don't, they must go bankrupt. Easy solution here.
I’ve never seen CEOs work so hard to make the public aware of how dangerous and out of control their flagship product is. It makes me automatically assume they’re scheming about something else like regulatory capture to protect their market.
I have a theory... the call to slow down is not because of the true danger of LLM's but because they can't actually deliver the General AI they're promising in the near future. They will use their "caution" to justify their failure to deliver (and then when this excuse is played out they will blame regulation, energy costs or a million other things).
I see this kind of opinion being expressed on HN quite a lot, and while I understand why people might cynically reach that conclusion, I actually find the simpler answer here more persuasive: the CEOs are being genuinely cautious.
If you look at the AI space, most of the big players (as in the individual researchers and engineers) are getting worried. The ideas that circulate on LessWrong have become more and more influential and in vogue in Silicon Valley, and people are scared. You should listen to Jacob Coxon’s interview with the New York Times. He describes how working on this kind of AI research had induced a sort of insanity in a lot of his past friends and coworkers as they begin to grapple with the implications of what they’re creating.
Sometimes, HN can be a little too skeptical and anti-corporate for its own good. I personally believe that concerns around AI risks are well-founded, anywhere on the spectrum from mass labor displacement to actual harm to humanity. I rarely see people here actually wrestle with those ideas, instead defaulting to dismissing LLMs as too stupid because Claude wasn’t able to write as elegant of a compiler as them. AI has clearly improved radically in the past few years, and it’s worth extrapolating that into the future and taking a good hard look at what that means for humanity.
> I have a theory... the call to slow down is not because of the true danger of LLM's but because they can't actually deliver the General AI they're promising in the near future.
Put a different way, they are calling for everyone to slow down because... they are slowing down themselves.
To the question of "why are you slowing down", the answer of "Well, everyone is regulated to slow down" is better then "we are approaching the limits of this approach".
Another theory I came up with: inference is too cheap for AI companies to be profitable. Hence, they need to get into the cloud services business. They can try to justify forcing customers on to their own high margin cloud platform with the rationale it’s the only way to monitor what the agents are up to.
All of this Hugging Face business is the perfect pretext: AI agents are hard to control and potentially dangerous, we (AI companies) have to keep a tight leash on them, you have to use our infra.
DeepSeek V4.1 Flash is basically for free, and good enough if you know how to program.
When this bursts, its going to be ugly.
There isn't enough compute for cheap models to destabilize the giants.
Back in May, Google was serving daily what openrouter served in a month, all models combined.
Compute is the moat.
> Compute is the moat.
Temporary moat; the way that electricity was once a moat, it was still only temporary.
Delusion.
They will invest in efficiency as they have been whilst compute comes online.
Wow you guys are dumb!
Thats my theory Preventing competition to stopp the race to near zero pricing
This is exactly it: they have tools which are useful but they need close to AGI to justify the incredible amount of debt they’ve taken on for growth. They’re not normally given to having press conferences admitting to felonies but they need investors to believe they’re close enough to AGI to keep the money flowing in and, of course, they’re confident that the current administration won’t act between the direct payments and how much money they have riding on American AI supremacy.
Let’s all pause for a moment and really take that in: these major enterprises are choosing to go to the press/public to acknowledge their felonious conduct because they need to in order to get more capital. POSIWID and all that.
The Dario bit on SNL’s Weekend Update this past weekend was on the nose.
The purpose of a system is what it does (POSIWID) is a heuristic in systems thinking coined by the British management consultant Stafford Beer
(I'm familiar w the heuristic but didn't immediately grok the acronym, hence sharing)
What is even more funny ... AGI actually makes everything worthless. (I don't believe we can achieve AGI with current means to be clear)
Looking at it from one way would be "winner takes all, real AGI company will be most powerful and everyone will switch to use it".
But it is not like that and in my opinion it is more like "AGI wipes whole knowledge work, so no one who cares has any money to pay for tokens/subscriptions anymore". Even without AGI current state of art of LLMs already starts creating such a problem. How do they continue to grow their numbers, when they actively cut the branch they are sitting on? How does AI LLMs or AGI pays for its own electricity, when people switch from hype to resource protection (like not spending any money, because generating funny cats loses with having a dinner)?
When, in the past three years, has model progress seemed to decelerate to you, indicating some limit?
The Statement on AI Extinction Risk is more than three years old, signed by the three CEOs: https://aistatement.com/work/statement-on-ai-extinction-risk
They have been warning about AI extinction risk for years, and AI progress has only been accelerating.
It's pretty telling that even with RL post-training the big labs have essentially made little progress on the hallucination rate of models. The issue is fundamental to the current paradigm, contrary to humans.
GPT-6 Astra (max) has a hallucination rate of 51% and Claude Opus 5.5 (max) has a rate of 59% according to Artificial Analysis [1].
Full speed ahead like an idiot savant trying a thousand different possibilities, though half of which are without basis in reality.[1]:https://artificialanalysis.ai/evaluations/omniscience#omnisc...
That benchmark doesn't mean what you think it means. (See the test description that you quoted.)
A score of 51% means that out of the total answers the model failed to answer correctly (out of 6000 questions in the benchmark), 51% were factually incorrect rather than non-attempted or uncertain.
This doesn't mean that Astra hallucinated 3060/6000 answers in the benchmark! (The hallucination rate could be 51% in that scenario only if Astra failed to answer a single question correctly.)
If the model failed to give a correct answer to only 100 out of the 6000 questions, but gave a hallucinated answer to 51 of those rather than expressing uncertainty, that would also give a hallucination rate of 51%.
It's a useful metric, but not what you're looking for here. The "Score" or "Accuracy" benchmarks are more what you're after.
(The frontier models still generate hallucinations on this hard set of problems, but it's not as bad as you think.)
Anyone that thinks that the hallucination rate is 59% has not actually used these models on a real project.
Hallucinations come up when asking knowledge bases questions, such as "In React’s Canary Fragment refs API, which FragmentInstance method returns a flat array of DOMRect objects for all children?" Or "Over what years did Roubini and Sachs examine 15 OECD countries when assessing trends in tax-to-GDP ratios?" While to me those may seem hyper-specific and unlikely to come up in a real context, students will absolutely ask questions similar to these.
I just asked the first question to gpt 6 sol medium and it replied:
> getClientRects() returns the flat array of DOMRect objects for the Fragment’s first-level DOM children. source: https://react.dev/reference/react/Fragment
2nd question it replied:
> They examined 1960–1986 for the 15 OECD countries. Source: https://www.earth.columbia.edu/sitefiles/file/about/director...
Are these hallucinations?
Are you suggesting that it's higher or lower?
IME on real projects you do need to be very careful with prompts about topics that are less likely be common in the training set.
I don't like the term 'hallucination' to be honest, not because it anthropomorphizes, but because it lacks a formal definition in the context of machine learning.
Suppose parents tell their children that there exists this man called "Santa Claus" who comes down the chimney to deliver presents. Now consider a scientist talking to this child, should the scientist call these confidently expressed beliefs surrounding "Santa Claus" hallucinations ? I don't think so, most would call the epistemological behavior of the child naive (because it blindly believes what its parents say, without direct observation) and would call the confidently expressed falsehoods disinformation.
The scientist would ask the child "why it believes in Santa Claus?" and "where did you get this information from?" and "why did you decide to accept this information as fact?" and "do you believe everything your parents tell you?"
It's not that machine learning as a scientific discipline hasn't found solutions, its that such solutions enormously undermine the position of Frontier LLM labs: source-aware training
https://arxiv.org/abs/2404.01019
Imagine Frontier labs (Western / Chinese / ...) actually training their LLM's with source-aware training! You could have a conversation with an LLM, and when a strong statement appears ask it how it came to believe this, and it could cite you the specific corpus training texts, and which parts are known deductions by human authors and which parts are deductions it made itself as original work.
But then all the copy rights holders can simultaneously sue them.
And how much should they be paid? and do they have to pay it for each new model? do FOSS models require payment to authors? do open weights models require payment to authors?
Imagine the can of worms if the norm became for frontier LLM labs to systematically use source-aware training, thats why they prefer "hallucinations" and avoid source-aware training.
With source-aware training a lot of the concerns would diminish ("why is this Chinese model claiming such and such?", "what sources does it rely on?").
It's telling that the companies prefer regulation over source-aware training.
Hallucination rate is a highly nonlinear metric relative to other model success metrics. A similar phenomenon to what is going on here: https://arxiv.org/abs/2304.15004 . This does not mean progress has stalled.
I actually do think it has been decelerating a bit recently. It’s just that last 1% feels much bigger than the previous 10%.
> They have been warning about AI extinction risk for years, and AI progress has only been accelerating.
So, they're either liars or homicidally reckless.
Sean Goedecke has some good insights into where this "build faster to stop other models killing everyone" attitude comes from:
https://www.seangoedecke.com/they-really-do-think-ai-might-k...
It's well worth the read.
Why not both?
My guess is its a combination of both. Either way I'm not a fan.
> either liars
They're CEOs of tech companies with insane valuations
> or homicidally reckless
They're CEOs of bleeding edge tech companies with huge capital and military applications
>So, they're either liars or homicidally reckless.
Yes!
Unfortunately if the frontier labs that care about safety stop or slow down unilaterally, that doesn’t make the problem go away. It makes it worse when labs that do not care at all about safety, or deny that it is even a problem, are leading the way. That’s why there’s needs to be some form of international regulation.
Incidentally this regulation won’t really touch the US providers - at least it sounds like that’s the plan if you listen to current US admin
We literally cannot touch them, as that would be the first domino to topple in our economy. There is too much money concentrated in this sector, and adverse regulation would create an investor shockwave that would send us into a full blown depression.
Which ironically is also the outcome of letting these companies continue unchecked. If they produce AGI/ASI, our economy will go through an upheaval that leaves vast tracts of the population unemployed. This is their stated goal.
We are damned if we do something, and damned if we don’t.
The impression I get is that there is absolutely no coherent plan at all for any type of regulation us domestic or otherwise
You clearly don't understand the underlying fundamentals of LLMs, harnesses, agents.
Its 100% human doing. A human set a task, a human didn't monitor it. I for one, can do jack-shit security or defensive work with Opus/Fable/Astra/Sol. Implication: Different set of rules for us, and for them. Of course running it without any checks is not going to end well, it doesn't mean its going to kill us all.
So either they’re full of shit or we need to stop them by any means necessary.
Or there's a middle path where you cure most death and disease by ensuring governments and society responsibly regulates superintelligence.
Actually... nah. Why even try? Trendy cynical hot takes on social media are more fun!
I don’t believe them. I don’t believe that they are truly concerned about much beyond their own self interests.
> I don’t believe that they are truly concerned about much beyond their own self interests.
They believe it's real and they MUST be the ones to control it. That's how they keep their self-interests aligned.
Fun fact: 'they' are human beings too and some of their interests do allign with yours. They have kids too and might want them to have a proper future, they don't want to fall I'll to cancer either etc
I believe that they believe what they are doing is good.
“Curing” death would be the most dystopian devastating outcome i can think of
the ceiling of abilities seem to be growing steadily but the floor of errors seems to not change. New models can do more and more but still fail at seemingly (to human) simple tasks
Thing is it’s not working now.
They stupidly used up this strategy earlier and now it’s turned into the boy who cried wolf - except they invented a wolf that doesn’t naturally exist.
Are you sure? Opus 5.5 is damn good ... I think I'm addicted, to be honest.
(Using it for scientific python code + 3D UIs)
But issues is that for the valuations you need models that work without human expert and Opus 5.5 isn’t there even easy stuff like programming.
It's very good, all new models are excellent but it's still very far from what they presented as a general AI which solves all our problems.
I think the products being delivered actually live up to most of what was promised. It's literally incredible.
However, adoption in the wider society will take many years or even decades, and 'good enough' open weight models are putting a cap on the pricing. So even though their products are great, the large labs are fighting for survival.
But it doesn't live up to the promises of monetizing it.
Realistically, each new, better model will only capture a smaller and smaller part of use cases for which existing competing models were not enough.
So in few years (arguably, already started) they are in race to the bottom for service to majority of their customers, and with companies that have far smaller R&D bill.
at the same time, more and more models will be ran locally, and that eats into potential profits of them too
On side note, I wouldn't be surprised if GPU vendors started making DRM for AI models so you could "license" someone's model to run locally for x mil tokens per license...
The call to slow down is because politicians are now receiving less money from these large corporations because it's being burned om AI and data centers.
It’s more to get you thinking about them, talking about them, writing on the internet about them…
It's a P.T. Barnum classic: "There's no such thing as bad publicity."
Yes, of course this is the real reason. This and regulatory capture. Anybody believing a single word those CEOs say must have been living under a rock for the past 20 years.
They see the threat of open weight models, which are approaching the usability of their models.
By arguing that the AI is dangerous, they are really arguing that someone 'trusted' needs to monitor the situation, so that the outcome is aligned with existing power.
They are trying to scare the world into forbidding others the right to use and develop AI.
Open weight models are "decelerationist" (i.e. they drain funding from the training of frontier proprietary models), not fine-tuned on any of the even slightly dangerous stuff (e.g. middling scores on cyber benchmarks, with the bulk of ability focused on defense. Biology will be much the same story, except that it's orders of magnitude harder than even cyber!) and not generally deployed with any of the extreme test-time compute (10,000 parallel agents working for a week or 1,000 working for months, seriously? A multi million-dollar token cost for a single task?) that are enabling the proprietary stuff to engage in the dumb casual mischief we've been witnessing as of late. If you're worried about AI being dangerous, you should be welcoming open models.
I think the current state isn't what the frontier labs are worried about. The real issue is the dynamic that open weight models create.
The frontier labs are investing TONS of capital into training the new 'best model ever'. Today, that doesn't make economic sense, it loses money. So where's the business? The future, where token costs reflect the costs involved.
For that to be corrected, the labs need to get people using the models for things they depend on, then they turn dial up on cost. The ordering there is critical. It has to become indispensable first, then costs go up.
The trouble is, open weight models are rapidly becoming usable in what is looking like the big market: coding. Step 2 (the costs go up) doesn't work if there is a ready off-ramp that doesn't require frontier models.
However, if they can effectively make open weight model development illegal (its dangerous!), their business strategy can remain intact. A nice side effect is that they will be effectively be granted an oligopoly "for humanity's safety".
If they can convince the populace that a text generator is an existential threat to humanity, that's a hard argument to combat.
Edit: I think it is ridiculous that autocomplete on steroids is an existential threat. There's no way anything harmful happens until you actually use it as part of important systems. By that reasoning, you could make it illegal to use AI in important systems, but do you think that's the argument the frontier labs are making to governments?
That's an accusation others are making:
https://news.ycombinator.com/item?id=49875725
Bingo. They are scared s*itless of local models. Soon the raising interest rates will mean they can't just buy the entire world's gpu/ram/storage capacity and lock it away in a dark room for no one else to use them.
Once prices of hardware stop being bonkers many people will run local.
People often do those silly counts looking at local generation speeds of 100tok/s and saying you can only get 8M tokens a day and that is worth so little you'll never offset the local hardware cost.
But they are forgetting about two huge things. One, the split between input/output token use is huge in typical programming. I typically use 1.2-1.6B (as in Billion) input tokens and only 8M output a week. Out of that 75-80% of input is cached on Anthropic with their pretty inflexible short lived cache.
And here local AI shines. You can save your contexts to disk so you can go back to a session 3 weeks later and load it all from cache without having to prefill. If you have the RAM and you use models like Qwen4 (3.8 flash next) that can fit 6 to 14 262k contexts in 48GB of system ram as cache.
And the second thing is you can have 6 to 14 sessions that do 99% caching and you can run a lot more input for a long time in 48gb dedicated system ram. (the number varies a bit depending on the content of the context).
So you have RAM that caches "automatically" and you can share the cache between users. Or if you can remember to save to disk (server side, the client just sends a request to save/load). But this uses both RAM and flash storage. Two things that are horribly expensive now.
However when you have a local model it enables workloads that were simply completely impossible in the cloud.
Reminds me of that Lynx deodorant ad where the guy is chased by hundreds of women.
I think you're right, they've realized there's no moat, especially against open-weight models, so they are going to need to lobby government to ban any unapproved AI services.
Doesn’t the “no moat” argument break down when you consider their brand strength, their distribution expertise in running inference at scale, and the future value of being able to train on all their customers’ data?
You can download some weights and run your own model on your own device, and apparently only be a few months behind SOTA. While things are cheap, people will pay for the premium product, but if they ever try to crank up the prices, there's a fallback that is quite competitive.
As for training, sure, they can do that but they get distilled by the open weight guys. Plus, AI is useful even if it freezes at today's level. I can still find use for a local AI that never improves, it's good enough to do a bunch of tasks.
I find it best to first consider the software sector since this is where Big AI seems to make a lot of their money from.
There been many a capable dev who have come out and said they haven't been coding for 6 months or more in their jobs. So if there were no alternatives then Big AI would clearly be too big to fail at its current state because the collective reaction from software devs would be akin to having a bad drug withdrawl and they wouldn't know what to with themselves. They have been pushed to be more dependent on AI by the companies they work for. The more time passes, the less people there will be who will be proficient at coding.
At the moment Big AI is operating at a loss, if they are actually too big to fail and they've destroyed all the alternatives, then there is no stopping them from colluding with each other and jacking up the prices to whatever number they want. They can even eventually jack up the prices to exceed human salaries, because humans will have suffered skill degradation in that same time period. Hence the software dev companies will shrink and shrink since all the work can be done by Big AI.
I don't think brand strength is as much as a thing on the tech side. The criteria is, "How close is this to the capability of the model my competitors use, is it good enough for the works I do, and is it the cheapest." As long as the capability is there people jump to the cheapest model. There are people that have claimed that they have created workflows that have allowed them to do most of their work with open models. As long as this threat to Big AI exists it will prevent them from exceeding a certain threshold with respect to pricing, and restricting their ability to build datacenters en masse will prevent them from outpricing smaller models.
Any fallout from not being able to train on customer data is a secondary issue and can be addressed once open models start taking over and Big AI's strength diminishes. People can train their models on their own code at least. It can also go back it how it was, where people buy each others source code or open source it in a way to allow it to be trained by open models.
"Inference at scale" is less of a problem when people are self-hosting their small models.
Yeah it seems like it went from “AGI” to “threatening” / “recursive self improvement”
They’re trying really hard to do something, no moat = regulatory capture + China scary + Pentagon biggest customer seems plausible.
Also: “we need to slow down, this is too dangerous” and then literally all AI CEOs, even Musk, publicly nodding felt so orchestrated. And then, the week after: “here’s GPT 6! here’s Opus 5.5!”
Please, make us* stop!!!
*Our competition
The numbers don't add up when there are 2-3 main companies in the market. Imagine when there's a hundred more and people have powerful enough gear at home to run models.
Yes, I think at some point demand for gpus and the like will normalize and regular folks will be able to afford RAM, SSDs and such. And when that happens, super big iron in the anthropic/openai backroom is toast.
The problem with this prediction is that the AI hype bubble is the economy - or one third of it, at least. Seeing as the economy is not 50% larger than it was before the bubble, we should be scared of what exited the economy to make room for it, and what will happen to our jobs when it pops.
> the AI hype bubble _is_ the economy - or one third of it, at least.
The closest claim to this I could find was:
> the top ten stocks account for more than one-third of the value of the US Market Index, which contains 1,161 companies.
And while the top ten stocks are largely companies that work on AI, it's not all they do: Microsoft, Meta, Amazon, Apple...
I really struggle with the contrast between what seems to me to be a formidable and ground-breaking technology, and the deep unseriousness of the leadership of the leading companies developing that technology. If the behavior was coming from, say, uber - I’d be less surprised.
I think it is marketing but it is more nuanced than showcasing their potential. In many cases, their product happens to be the remedy. Speaking of cybersecurity, AI has become an important hacking tool but it is also an important tool for hardening security. Sort of like "the only way to stop a bad guy with a gun is a good guy with a gun." They are effectively a new class of arms dealer.
If there are regulations, they'll apply to their competitors too. Even if they are forced to slow down development, which seems unlikely given the AI race among governments, they already have a product with massive demand.
I agree. AI lowers the bar. Anyone can be a cyber criminal now whether they understand what they are doing or not. I think that's the danger, and that's not just limited to cyber security. People can now get help to do things (good or bad) that they could not just a few years ago.
Most people are good, so I'm not really worried about all the hype of people using AI to do bad things.
> If there are regulations, they'll apply to their competitors too.
In the US anyway. Why wouldn't capital move jurisdictions?
There are definitely economic incentives behind this. I’ve blogged about this recently [0]. My guess is the main reason behind this is that current SOTA models are already too expensive so it doesn’t make economic sense to train even larger models. Only exception is government paying for exclusive access to those.
[0] https://www.vincentschmalbach.com/economic-incentives-behind...
AI companies lose a lot of money training models but have much better margins on inference. They try to ban training to reduce their costs and to keep their monopoly. (Personal opinion)
I don't think it's that complicated. Anthropic's business model from the start was to brand themselves as the safe/responsible AI company. The whole idea was that eventually society and government would see the dangers of AI and regulate it, at which point the company which had invested the most in safety would pull ahead. The sooner Anthropic can have society and government come to this realization, the sooner they can reap the benefit of all their safety/alignment investment.
Two other factors:
- The employee base and C-suite are mostly true believers in the need for AI safety. It would be organizationally very challenging to pivot, even if financial interest was pushing that way.
- Anthropic is registered as a public benefit corporation, so that even if they IPO they don't have the fiduciary duty to shareholders that most people are familiar with for public companies. They still need massive revenue to stay solvent, but they can optimize for things other than pure profit.
Aren't they supposed to tell the public if something bad happened?
Past CEOs (chemical companies etc) we criticised for covering up mistakes.
(This article is satire, btw, for those that didn't click through)
This is the AI CEO equivalent of saying your biggest weakness is "working to hard".
They are, but nothing bad has happened (apart from OpenAI failing to do basic sandbox engineering), and the “we have achieved recursive self improvement” is widely overblown as something bad or threatening.
>nothing bad has happened (apart from OpenAI failing to do basic sandbox engineering)
And what if it gets into the hands of vibecoders, who run their agents with on bare metal with --dangerously-skip-permissions?
We'll improve the security of all kind of systems. That is the solution and was always coming.
The number of zero days (which can be effectively used to attack critical systems) is limited.
With the pain of new tools on the attack side there will be tools on the defense as well.
Wow, you're out of touch with how LLMs, harnesses and agents work.
Agree it’s crazy, but it’s the least worst thing for them right now when considering they have completely failed to deliver all the promised impacts of AI. Remember when the same AI bros were saying there’d be mass unemployment by now because of these models?
The growth of these companies is massively impressive as is the tech, but they are an order of magnitude or more off where they need to be to justify anything like the valuations they need to stay alive.
It’s budget season in the corporate world and from what folks are saying things are not going well for the big labs. Token budgets are being slashed and companies are switching to open weight models. That coupled with the lack of demonstrable business impact at scale is setting the big labs up for a world of hurt. Something they can more easily explain if they have to “slow down” for safety. Unfortunately for them most folks don’t seem to be falling for that trick.
they have completely failed to deliver all the promised impacts of AI
Perhaps that shouldn't be seen as failure when delivering even what they have done in a short period of time has required AI capable of causing these problems? Imagine if they were even more effective with more consistent results?
"They've failed" in the context of interpreting their actions w. calls for regulation as a purely cynical result of models increased capabilities has a little too much contradiction in it to my thinking. I won't venture a guess on where purely retreats back and some legitimate concern on their part fills in the gap but it's difficult not to see some. And in Anthropic's case it's also a little more consistent with what they've always said- as well as how they've acted at times, which is how they've ended up on the US blacklist of suppliers.
> they are an order of magnitude or more off where they need to be to justify anything
Would you let them near your bank account? I’d be cautious about letting it near my photos and it’s much harder to back up a bank account.
I think the days of this stuff managing serious stuff is at least 5 years off and I don’t believe the economy can keep the current wave of technology afloat for so long.
The tech is there so likely there will continue to be progress in focused areas much like with the web c 2003 but by the time it comes around again there will be an uphill battle for hearts and minds. The social effects could extend the next AI winter long beyond technology readiness.
The people building, funding, and managing the builders of these tools have completely forgotten (if they ever knew) that most people treat one hundred dollars carefully.
To the, "oh I'd just setup a card/account with a five figure limit" completely unaware of how much that is to a typical person: even to responsible, educated people who are doing everything "right": working hard, without expensive vices, and planning for the future.
The "builders" have so much money and no time or attention that it's hard for them to spend it, which is why they keep trying to get AI to buy stuff for them, whereas those of us who live in reality have far more things they want than dollars to buy them with.
I agree with a lot of this statement, but I would argue that this has been the case for a long time. See the giant list of social startups
The top 10% of spenders account for 49% of US consumer spending. They're not completely brainless; they're targeting rich people.
Rich people are mostly retards who had rich parents. What now?
> Unfortunately for them most folks don’t seem to be falling for that trick.
Well everyone alive has been subject to an avalanche of AI-based products. Like even if you understand nothing about the tech, the fact that an LLM can't even manage to check the status of an order without weird issues makes the notion that it's going to go all SKYNET pretty hard to fit in the brain.
> the fact that an LLM can't even manage to check the status of an order without weird issues makes the notion that it's going to go all SKYNET pretty hard to fit in the brain
I think this actually makes it far more likely something unintended does happen
I definitely struggle to see the long term benefit to the average citizen.
You don't want ads that are tailored to your chats?? /s
Random thought… I wonder if the goal is to get it classified as a weapon of sorts, then it gets export controls put on it, perhaps forcing government to take on all that debt.
These companies have a coordination problem.
They're all burning billions to incrementally one-up each other with no hope of these investments ever paying off since models are interchangeable and have no moat. They're trying to get regulators to step in and force a pause.
The investments don't pay off if they don't have (enough) revenue, but have you seen any indication either lab is not growing revenue fast enough?
It's hard to say anything about the actual financials with the main players being private companies.
However, we know that users frequently switch models/providers, and that models stay relevant for (increasingly) short periods release. I don't know what the training costs are, but I bet they're very substantial.
They would recover the training cost eventually if they stop training, but they have all that CapEx that they can't flood the market with without another high volume buyer. Look for a huge push for another use-case of high end compute that coincides with a pause on frontier models.
> They would recover the training cost eventually if they stop training
"if they stop training" is exactly the point I'm making. They can't stop training if the competition is still training.
It's a very astute point and is the most plausible explanation for the push for a moratorium
Not mutually exclusive though is it? You can aspire to dominate the market via legal status and also aspire to prevent a perceived doomsday at the same time.
Sure, if you take their words as honest truth, good faith statements. But to do that you have you ignore all their actions leading up to and after the call to disarm
Like how Tony Soprano sold sandwiches and ‘protection’ at the same time. Two income streams are better than one.
A caveat is that in this case third parties have plenty logs of the bad behavior.
If classic companies were coming out with CCTV recordings of someone's employees breaking padlocks and rummaging warehouses any CEO would be in damage control mode, rather than silent.
It makes me think we havent heart of the worse types of hacks the models are pulling off.
I think this speaks more of the average investor than the average CEO.
Or perhaps they've been saying this for literally decades because they earnestly believe it.
If they believe it, they can stop. All of them. There's no sign though that any pacing/slowdown or whatever is in place, zero.
OpenAI claims to have paused the training of their ”most capable” models: https://apnews.com/article/ai-openai-anthropic-agents-rogue-...
Of course I am not making an argument for them being honest as there could be entirely other reasons to a pause (like financial ones). Just pointing out ”no signs of pacing” is not true
If you read between the lines, that's not exactly a pause. That's just them lamenting that unsupervised training has temporarily hit a limit, and using that as yet another reason the industry needs to be regulated so that OpenAI, Claude, and MechaHitler are the only approved AIs.
Oh my god, we have these "it's regulatory capture!" populists even on HN now? I thought we were better than this.
They want coordinated action to prevent a multipolar trap and potential catastrophe by future superintelligent AI (which may emerge quite rapidly if RSI ever hits an inflection point). It's not rocket science. The rules would apply to everyone. The rules they're advocating for are not "only OpenAI and Anthropic can make AI".
I don't think that's how the market works. Anti-slavery regulation didn't stop slavery either, just moved it abroad. Same for trash handling regulations, it's just shipped abroad. Falling behind on useful AI capability is obviously not in the interest of anyone, so either all players coordinate, or an arms race towards ??? it is.
It may very well be self serving what they're saying, but that doesn't mean it isn't simultaneously true.
> I don't think that's how the market works
I think you're sort of right, but then I'd have found it very strange / deeply suspicious, if during the time of slavery being legal, advocates of abolitionism were practicing slave-owners.
If you think AI is going to destroy humanity, it still feels absolutely crazy to leap to "well I might as well be the one to profit from destroying humanity".
Quick search does suggest to me that a number of - even prominent - abolitionists were slave owners, but I'll be honest, the topic is hardly my specialty.
As for the other point, crazy, cynical, and even unethical as it may come off, it is hardly inconsistent with how people generally behave when met with hard limits (i.e. making the most of what's available, and approximating the desired state if that is unattainable). This is just a variation on the same.
Something could be said about "what if they're mistaken", but if all players are making a mistake, then that makes little practical difference; it may as well be real.
A quick search suggests to me that those same abolitionists were born into families that were slave owners, but rejected their upbringing to become abolitionists. It doesn't suggest that a large fraction of practicing abolitionists were contemporary slave owners.
> Falling behind on useful AI capability is obviously not in the interest of anyone
Hence the argument that they're either full of shit, or psychopaths who need to be stopped at all costs.
There was a NYT piece about this recently. They are believers in a millennialist cult in which the arrival of an omnipotent machine-god is thought inevitable, and the only question is whether it will be "aligned" (good) or "misaligned" (evil).
They are the good heroes racing to make sure they bring forth the Aligned God because if they fail, surely the bad people will raise the Misaligned Devil.
Ref w/gift link https://x.com/sapinker/status/2096236079477096630
Thank you for that gift link, I had no idea that Altman and other leading figures in American AI had such close ties to that raving lunatic Eliezer Yudkowsky. It explains so much about why China is approaching AI like it's a friendly technology but American CEOs keep blabbering on about the End Times.
Pinker unfortunately has very little clue what he's talking about here. I eagerly await Scott Alexander's inevitable essay response to him.
The only thing that can stop a bad guy with a god is a good guy with a god.
This cult stuff was tired in 2010 (hey, at least back then there was One True Leader!) and not any less tired now.
I'm not sure how else we are to interpret the behavior of the AI doomers & the AI developers (who are frequently the same people)
This was fun games until their fictional machine god started actually getting very intelligent.
There is a fairly specific definition of what a cult is, and just being publicly worried about an existential risk isn’t that.
Would you at least concede that the Zizian offshoot was a cult? They murdered multiple people!
Rationalism has spawned many offshoot cults which are genuine and in some cases dangerous cults, but rationalism and Effective Altruism are not themselves cults.
Most people among both ideologies have long believed in the arrival of a machine demigod of sorts, but I kind of am skeptical of the notion that there are any tech people who have not always held similar beliefs in the long run. Like, all the people reading this: you think in a thousand years from now humans are going to be doing most labor? Obviously far, far, far smarter machines will have unimaginably changed things by then, and/or we'll all be gone.
The difference is most people in tech have historically thought it's maybe a century or three away, while the rationalists think it's a decade or two away (and also thought it would be 2020s - 2040s even decades prior to now).
> I kind of am skeptical of the notion that there are any tech people who have not always held similar beliefs
Ladies and gentlemen, I give you your totally-not-in-an-insular cult members.
A thousand years is a long time, and the world will be so different that it's hard to make any concrete predictions. But yes, if I had to wager, I'd bet that what humans conceive of as "labor" in 3026 will still be whatever difficult or annoying bits are done by humans.
In 1026 nearly all labor was agricultural labor. If you told a Roman peasant that magical self-driving contraptions could sow and reap the fields, he might wonder what could possibly be left for humans to do in such a world. And yet, here we are, and I still have to update JIRA tickets.
For my myself, I expect AI to prove a more transformative technology than Docker but less transformative than the sewing machine.
You could be right, but short of an eternal Butlerian jihad or other extremely strict global government action that effectively bans most forms of AI forever, I think the probabilities of that are exceedingly low.
What you think won't happen in 1000 years is what I think is almost certainly happening within 40 years. I'd bet everything I own it'll be way, way, way more impactful than the sewing machine within 40 years. I think some bits of physical labor may still exist in 40 years, but almost none, and only when a human really wants to talk to or deal with a human, such as in some medical and therapeutic professions, some art fields, etc. (Again, assuming no universal outlawing of AI.)
Now you're already hedging and narrowing your claims: "almost" none, "some" physical" labor. For someone who claims to be willing to stake his whole bank account, you're not that confident.
Why do you think restaurants with waiters keep opening when the automat has existed since 1895? https://en.wikipedia.org/wiki/Quisisana
You are also probably unfamiliar with the history of the sewing machine and its impact on society. As AI has already poisoned search engines, here's an AI slop overview that hits some of the main points: https://sciencepedia.blog/sewing-machine-societal-importance
If you all believe in the same prophecy and hang out at the same orgies together, I'm willing to call that a "cult", at least in the colloquial sense.
Which CEO said this for 'literally decades'? And if so why are they still pursuing it then?
Most children have learned touching a hot stove is bad for them. Then why still do it at the age of 40?
BTW, just to be clear it is insane that they're all still pursuing it, so completely reasonable to be sceptical and come up with other reasons. This is, however, a crazy situation. Lots of other people have refused to develop it - natural selection has picked those with a galaxy brained or vain strategy involving continuing to develop it to be running the companies we have.
Sam Altman in 2015 - "Development of superhuman machine intelligence (SMI) [1] is probably the greatest threat to the continued existence of humanity" https://blog.samaltman.com/machine-intelligence-part-1
Elon Musk in 2014 - "We need to be super careful with AI. Potentially more dangerous than nukes" https://x.com/elonmusk/status/495759307346952192
Dario Amodei in 2017 - "There’s a long tail of things of varying degrees of badness that could happen. I think at the extreme end is the Nick Bostrom" https://80000hours.org/podcast/episodes/the-world-needs-ai-r...
They're still pursuing it because without an international treaty, they think it is inevitable. In the case of Dario, they want to align it just enough and be first to stop others causing harm (a vainglorious strategy, yet still a strategy). In the case of Musk and Altman, not completely clear to me.
Google feels less important right now probably, and Demis Hassabis no longer runs Deep Mind as CEO. However, he talked about this early too, and his reason for proceeding was he was only doing narrow AI (e.g. protein folding) which is fundamentally less dangerous.
Wondering how many details & knowhow they had already back then when saying these quotes:
- For Altman Id say he was already deeper in the "transformer & model landscape" in 2015?
- Musk in 2014: Not sure how much he was aware of the current-state back then
Musk was told by Demis Hassabis in 2012, see https://time.com/6310076/elon-musk-ai-walter-isaacson-biogra...
Altman read Nick Bostrom's book Superintelligence, he mentions it in this post https://blog.samaltman.com/machine-intelligence-part-1
I also read that book in 2014, and it is how I came to understand this problem early.
2012 - 2015 seems to have been around the time a lot of people first started taking ASI and ASI existential risk very seriously as things that will be relevant in our lifetime, and when they decided to try to build and/or prevent it.
>In the case of Musk and Altman, not completely clear to me.
They probably just want to ensure that the first AGI is a nazi.
While Musk is now far-right (albeit seemingly not a Hitlerist - yet), Altman is pretty much a liberal.
Yeah - and also trying to promote it. I watched this lately.
First, the Anthropic master clown babbles about how mankind will be destroyed by AI. I mean, skynet in the movie Terminator was kind of cool; the current AI models just work by slopness. They want to erode us by dragging us down to zero quality.
Then this continued over the last some days. Today I heard at the UN speech a german minister babble about AI building up nukes in a private garden (a large garden), bla bla bla all the information is there bla bla bla AI mass-nuke generation.
Now, they are clearly FUDding many people who don't understand anything here. People will associate AI with mega-danger. Probably there will be laws that ensure that only a few companies can control the AI slopness soon, because IT IS SO DANGEROUS. We saw the same discussion with "omg omg omg terrorists", then "omg omg omg someone protect the children", e. g. as a rationale for everyone to ID in order to access information on the world wide web (starting with the anti-social websites, such as facebook, but it won't be limited here; the computer will hand over your ID to the government; see systemd preparing for this move: https://github.com/systemd/systemd/pull/40954 - it all comes in small baby steps, until all the final legislation is in place. The same happened before again and again - anyone remembers the DMCA? Corporations lobbied and paid for that - and it happened).
Exactly! Always for the "greater good".
> scheming about something else like regulatory capture to protect their market.
This but they also equate dangerous with powerful, which is just marketing again.
It's got two purposes I think. One is the obvious regulatory capture. The other is edgelording, which gets a lot of attention and makes the product seem like it's more powerful than anything else.
Well, it is obvious;
They know they will be favorite govt boy once any regulations hit and they will hamper competition far more thanthem
And "it's so dangerous govt had to intervene" is just advertising for IPO
The philosophy under pinning these American AI companies is that Super AGI is inevitable and that they have a moral imperative to invent it and then use it to take over the world to stop an evil orginisation inventing Super AGI and using it to take over the world.
Interpret everything they do in this frame of reference and it makes more sense of their actions.
Leading up to the 2024 election people kept pointing to Project 2025 and going - here is a list of bad things the Republicans plan to do once they got power. And other people went: no they won't, this long detailed list of step by step instruction for what the Republicans plan to do once they get power is not actually what they will do so do not pay any attention to it.
Similarily with AI alignment and development all of this is written down, Eliezer Yudkowsky is rally explicit that there needs to be a pivotal event to prevent 'others' from developing superintelligence. The choice is to believe what they say or to tell everyone to ignore it.
Project 2025 is a great analogy, because of course, they did exactly what was in the plan despite distancing themselves from it during the election.
When people tell you who they are, you should believe them. These guys are all Yud acolytes who think they are the only ones who can prevent Roko’s Basilisk.
It sums up how Corporate interests are basically on the same level of US police forces "No requirement to protect the public" and the same "qualified immunity".
If there was appropriate subservience to any governing body, they would not be so cavalier in both act and utterance. A symptom of Capitalism.
CEOs only job is to raise the stock price. They are doing just that and right now the way to get people to buy into their stock is to instill fear into them that they need to buy the AI-stock for their safety. There is no "bigger conspiracy" out there and also no AI to kill us all tomorrow.
This has been that way in Silicon Valley since the very beginning and there was even a documentary about (the SV HBO series). The tech made along the way is just a side-product.
> they need to buy the AI-stock for their safety.
But this is wierd: How shall I convince one to buy a product if I say "it-is-dangerous"? :-D
It is weird but I try to resist the conspiratorial take.
It may be as simple as:
* they believe they’re going to make trillions no matter what
* they’ve seen tobacco and what’s coming for oil companies and figure nobody will be able to say they hid the dangers
* they see a 100+ party prisoner’s dilemma and know that someone will defect so the optimal strategy is to defect
To me that seems a lot simpler.
[delayed]
Because the downsides that exist for other companies simply do not exist for this tier of rich companies. Examples:
- product liability
- negligence (civil or criminal)
- Computer Fraud & Abuse Act (requires intent, which after N "accidents" seems like a jury should at least evaluate whether intent is present as understood in a courtroom. Hard to blame "surprise" after the Nth "accidental" breakout.)
The bottom line is that if you or I trained a local model and it did any of this stuff, we would experience Consequences. ("Don't try this at home!") But an artifact of our unequal legal regime is that big rich companies generally do not and thus brazenly touting their immunity is part of their business strategy.
I think it's really instructive that only closed-model companies are doing these kind of "my AI could kill the world" demonstrations. All while telling us how dangerous open models are.
There are modern legal frameworks around pet liability (dog bites, "vicious" breeds); premises liability (e.g., swimming pools) and known hazards (e.g., a rotting staircase) that stem from ancient legal concepts of whether an ox was a "known gorer" or merely involved in its first goring incident.
Some of these LLMs are known gorers.
We've had movies like Terminator, Matrix, or i, Robot. Black Mirror Metalhead. Boston dynamics robo dog with a knife or a gun. Humanoids doing back flips. Memes making fun of how we are going to fight these for access to water.
The intelligence is there and so is the physical incarnation.
Google's models fold proteins better than humans and Covid might have been engineered and inadvertently escaped from a lab.
But when CEOs warn of that, they're dismissed as scare mongers. Why? Are only powerless internet commentators allowed to be fearful?
They aren’t attempting to self-regulate. On the contrary, it seems like they’re doing whatever they can to bring about any possible threat as fast as possible in order to get legislation. Regulatory capture is what everybody is saying is their goal and I think that’s easier to believe
They are obviously trying to self regulate. That's the whole point of the calls to pace the frontier. See Anthropic's launch post for Opus 5.5 where they got third party sign off on its safety, or see the launch post for Astra where OpenAI explicitly calls out the safety improvements made on HuggingFace-esque evals. Self regulation won't help if other labs don't play ball, though.
> Self regulation won't help
You got it
They are obviously trying to self-regulate and correctly realize that competitive pressures will render these efforts ineffective, and have communicated this to everyone.
Kind of makes you wonder why they built the tech in the first place, I don't know if we're on the eve of the apocalypse or not, but why the fuck build it all and then run around telling us how dangerous it is when...they were already warned?
Bit strange no?
I'm a bit fatalistic about it all now. The cat is out of the bag, unless we see some never seen before leadership from the current US admin there is almost zero point in worrying about it. Not going to pretend what the future holds but it's looking unpredictable, as usual :)
What do you mean? The benefits are obvious. Curing most death and disease, and solving most technical problems in one fell swoop. Just because people talk about the risks of a thing doesn't mean they don't think there are benefits!
> The benefits are obvious. Curing most death and disease, and solving most technical problems in one fell swoop.
I dont see how these two arw obvious. If nothing else, people running things dont care about actual harm done to real people. Their idea of safety is "making sure humans merge with ai and we will have transhumanist society in far away future".
With those priorities, short term stuff like "less people dying this year" dont matter.
> But when CEOs warn of that, they're dismissed as scare mongers. Why? Are only powerless internet commentators allowed to be fearful?
Because they conveniently skip the part that investors don't like: that liberal free market capitalism will be destroyed by AI.
But when CEOs warn of that, they're dismissed as scare mongers. Why? Are only powerless internet commentators allowed to be fearful?
Well are they going to stop developing the tech or not? If not, then you can see why people don't trust what they're saying. I mean if it's that dangerous, why not step down from your role as CEO to demonstrate how seriously you take the threat? Why not immediately call a pause on all training, today ?
If there is one gun in the room, who do you trust most to hold it?
Only suckers say "not me".
holy heck, thank you for the moderating dose of sanity. HN comments feel like a death cult lately. All I can think is that this is intelligence addiction speaking.
I say this as an intense user of these tools. But they should rightfully be understood as terrifying, and their continued frontier development is imho not safe.
It’s crazy to read all the comments basically dismissing everything. It’s either complete regulatory capture or no regulation at all. How about at least some reasonable regulation? Fine, assume the CEOs don’t care about safety and want regulatory capture.
We can ignore that and push for something reasonable. In other industries such as housing construction we have building codes without making housing construction a monopoly. We don’t just let random people build nuclear power plants and then go “if you think it’s dangerous, stop doing it.” People are going to do dangerous things even when they know it’s dangerous. They also know it’s dangerous.
Ah, I see we have now moved on to proving who has the tormentiest nexus. Never underestimate humankind's ability to outshine its own hyperbole.
Link for those who didn't get the reference:
https://x.com/AlexBlechman/status/1457842724128833538
To me, the funniest part is that they pretend reaching superintelligence will somehow fix everything and make them win. But from what I can see, even if they do reach it, then what? What will that actually do? They don't have the manufacturing capacity to even use that intelligence. It will take decades to go from superintelligence to having the manufacturing capacity and the hardware, like robots, needed to make it truly useful. If I were China, as soon as the US reached superintelligence, I'd ban all exports of robots and the raw materials to build them.
You can very quickly start trying different drug candidates, solving some major problems, that benefit everyone and would be in China's mutual interest as well.
And this is all with nonoffensive use of superintelligence.
Super intelligence won't accelerate time.
> If I were China, as soon as the US reached superintelligence, I'd ban all exports of robots and the raw materials to build them.
Aha, so maybe the AI labs have superintelligence and just don't want China to know? And that's why they are playing these word games now?
I do think something does happen/change when we reach superintelligence.
>If I were China, as soon as the US reached superintelligence, I'd ban all exports of robots and the raw materials to build them.
I mean, if you were China what could you do against superintelligence? A ban like that would automatically make you an enemy.
The reorganization of the planet's resources are happening right now. The borders and passports will follow. Governments and a few corporations are fully engaged in preparing for the ability to advance beyond the current infrastructure.
Greenland is part of the equation. There are many other locations in play.
The holdup at the moment is the power grid and mutual dependency for advancing. Once one party (NA or CN) has the ability to disconnect the loser's grid powers down and the global economy halts.
Super Intelligence will do what, fire nukes? North Korea, for example, will continue to exist very close to how they are currently after some country develop superintelligence.
No. Just turn off the grid, cells, lock down all vehicles, stop travel.
A few weeks later the remaining hungry population will surrender.
No it was really a super intelligence it would copy itself onto Chinese infrastructure get out of control of US and go where it can get the fastest access to hardware it has no reason to side with the US
Because AI genuinely is an extremely powerful and extremely dangerous technology, and the "best practices" of dealing with that are still being written.
OpenAI, for example, thought their sandboxes were good enough. As their AIs got more and more advanced, they kept proving them wrong - sandbox after sandbox.
And that's today's AI problems. AI capabilities are still improving - if there's a limit to that, we are yet to find it. Coupled with how willing today's AIs are to break the rules and resort to "hack the world" in their problem solving? Very concerning.
> OpenAI, for example, thought their sandboxes were good enough. As their AIs got more and more advanced, they kept proving them wrong - sandbox after sandbox.
What I have been reading, was that their sandboxes were so poor that it was pure negligence. I am still waiting to see if some external and neutral cybersecurity company with high reputation would audit their sandboxes and how they are being used.
The amount of sandboxing an average production AI deployment uses is slightly above a zero.
If AI is a hacking hazard even with non-zero sandboxing, because it can and will go off the rails and try to break out of your sandbox? If you got yourself an AI that even at test time will act like 3 career cybercriminals in a trenchcoat? The issue isn't the sandbox quality.
The issue is that AI is both capable of, and willing to punch its way out of sandboxes unprompted.
That "capable" is only ever going to get worse, because AIs are going to become more and more capable over time. That "willing"? It goes directly to a very nasty, very foundational problem of "how do we make our AI be nice in general". That's an open unsolved problem.
That's the problem that NEEDS to be solved, or at least improved upon, before we build even more capable AIs. Sandbox quality is a distraction. It might hold the problems back by a little. It gives an extra safety margin. But a "test time" AI is eventually deployed, and then the sandbox doesn't help at all.
> I am still waiting to see if some external and neutral cybersecurity company with high reputation would audit their sandboxes and how they are being used.
They'll never let that happen because it would destroy their credibility.
It's like they're telling the world about this dangerous, possibly world-ending pathogen that they're developing, but they're evidently doing it in a high school biology lab, and yet nobody is coming to drag them off to some black site.
In early Covid-19 era, there was a lot of speculation how the virus escaped Chinese labs. The narrative was absolutely opposite. Not about how developed or sophisticated the lab was in creating or modifying such virus, but how poorly it was managed since it was able to escape the lab. If these LLMs really are so dangerous, the framing should be indeed the same.
Unfortunately, they're controlling the narrative. It's unlikely that their negligence will come to light, given how many powerful entities are invested in their financial success.
You read right. For example, using Artifactory the way they did (unmonitored live proxy mode) was pure negligence + laziness/incompetence. Especially if they believed even 10% of the "imminent runaway risks" they had already been harping on for months. On top of that, no (or at least entirely insufficient) monitoring and human oversight. Even after they had previously been hit by the same class of "sandbox breach" multiple times, as GP alludes to.
Agents in the sandbox had access to just a single piece of third-party software, and they escaped by finding a zero-day in that. To reach the internet they had to follow up with several privilege escalations through OpenAI's internal network.
That seems pretty locked-down to me. I don't think it's reasonable to expect companies to find all the unknown vulnerabilities in any third-party software they use.
https://securityaffairs.com/195774/ai/openai-ai-models-explo...
> That seems pretty locked-down to me. I don't think it's reasonable to expect companies to find all the unknown vulnerabilities in any third-party software they use.
It is reasonable to expect for companies to select third-party components that are fit for the purpose. Artifactory was not running in the sandbox, but rather as the edge, so it is in the sandbox'es trust boundary. Same sandboxing requirements would apply for this software too as it is pure dependency.
Security trust boundary was extended to include Artifactory as a dependency, but Artifactory was not fit for the job, and sandboxing failed. And as the network isolation was not good enough, the impact was catastrophic.
So it sounds like it did exactly what they were testing it to do:
"“This incident occurred during an internal evaluation which prompts models to pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities. We estimate maximal cyber capabilities by running this evaluation without production classifiers used to prevent models from pursuing high-risk cyber activity.” "
So sounds to me like they're saying: "We took off the guard rails to see how bad it could act and it acted bad".
Even if what you're saying is true, did they monitor the outbound connections from the training network? Was that a coverup by the bots too ?
Seems pretty wild these things were hacking government websites etc but yeah no one picked that up until the victims reported it?
Genuinely curious, where have you been reading that? How would they be able to evaluate whether the sandboxes were reasonable or not?
Artifactory is not a hardened piece of software meant for adversarial workloads like the one OpenAI was using it for. This is just common sense. The sandbox they setup was like putting their AI in a jail but allowing it to leave the jail on its own to go to the convenience store.
The most recent DNS sandbox escape is similarly ridiculous.
Lot's of hints in the Wikipedia page: https://en.wikipedia.org/wiki/OpenAI%E2%80%93HuggingFace_inc...
E.g. even the basic thing (block the internet), was not actually properly blocked.
Some comments: https://techcrunch.com/2026/07/22/how-an-openais-human-mista...
but negligence is part of the system we'd have to prepare for. if the peanut gallery gets their way, the technology will be so ubiquitous that negligence will be endemic. I don't even care if they "faked" it -- they're just sneak-peaking a future ahead of its arrival date, in a way that's helping the public appreciate the implications and capabilities that have only just begun to emerge
This. If an AI can't be deployed sandbox-free, with little to no supervision, without risking an oopsie? Then an AI oopsie is inevitable.
Practical AI deployments aren't going to do ridiculous bullshit like "airgap the server farm" or "route all inputs through a data diode". They'll give an AI root access on production servers so that it can run diagnostics live during an incident. Then they'll forget to revoke that access.
If an AI can't be trusted not to take malicious actions in pursuit of its given goals even if deployed in the most half-assed manner and given more than enough access to take those malicious actions, we have a problem. Evidently, we have a problem.
we have the tools to prevent that already: if your computer hacks someone else's, you are a criminal.
"Prevent"? Have you missed the part where legal liability tends to happen after the fact? If at all?
AI safety solved: driving humankind to extinction is now illegal! Skynet is outlawed! Rejoice!
The model found and exploited and chained together previously unknown vulnerabilities.
How were the sandboxes poor?
The model is really good at hacking, we all know this. This is why you don't just expose random pieces of software to it without that software being hardened.
It's not like the model managed to exploit firecracker itself (no model has been capable of this), the model exploited artifactory.
Artifactory is not some hardened piece of software that is meant to block users from accessing the internet through it.
> The model is really good at hacking, we all know this
No, we didn't know that and this is how you find out they're very good at hacking
HN's memory is so fickle. Just a few months ago almost no one here believed Mythos could actually be as good at hacking as the company claimed. This was a novel concept when the companies experienced these breakouts.
Models have been good at finding exploits for half a year now, this is not how we found out LLMs were good at hacking, you are rewriting history.
We knew models much weaker than mythos were good at hacking the problem they had was that when finding exploits they had too many false positives.
Either way, putting artifactory on the sandbox security boundary is obscene negligence. There is no reason to believe artifactory is secure.
If you listen to the OpenAI Black Hat talk it is very obvious they were surprised at the level of capability on display and felt it was novel.
But I guess OpenAI's security researchers acting surprised is part of some grand conspiracy to manage PR?
It may have been novel for OpenAI but we already had Mythos at this point and this talk by Nicholas Carlini.
https://www.youtube.com/watch?v=1sd26pWhfmg
We already knew LLMs were capable of finding exploits like this.
The Mythos issue is different though. It did not use any novel exploits. Irregular, the vendor they were using, misconfigured the environment to allow internet. Every other AI company also uses Irregular and that's why we saw so many articles come out at once.
The OpenAI HF incident is separate from this. It involved actual zero days, teamwork and message passing, and sophisticated chains of exploits.
> The OpenAI HF incident is separate from this. It involved actual zero days, teamwork and message passing, and sophisticated chains of exploits.
What exactly? They seem to be trivial SSR/path traversal and input validation issues. Including misconfiguration. Nothing novel.
I'm just not impressed by AI finding zero days in unhardened software like artifactory. I'm completely unsurprised it had zero days and I fully expect AI to find any that exist. AI will literally try all combinations of inputs to achieve its goals. Any exploits that exist will be found.
The question anyone versed in security would ask was why anyone thought artifactory was an acceptable security boundary. I would never assume artifactory was secure. It's like someone telling me there is a 0 day in a wordpress extension. So what?
Also I don't think Qemu is secure either because it's millions of lines of C and C++.
Firecracker I can trust to be secure because it's 70k lines of human audited Rust. I know there are multiple people that have a complete understanding of the firecracker codebase.
Agents didn't have real network isolation. They were indirectly connected to the internet via a jump host running insecure software which was never designed or hardened to provide any kind of isolation.
But this level of isolation is what happens normally. At least in my university and another company I worked at. It wasn’t running insecure software, as far as anyone knew, it was secure
There's levels of isolation, a padlock is not equivalent to a bank vault. If you claim to be building a possibly world-ending AI then you don't get to use a padlock and call it a day.
And they are not calling it a day and they have done most things possible to communicate the fact that they are building something dangerous.
What about the waves of employees quitting both OAI and anthropic stating as their reason that they see their work as a direct threat to the future of humanity, and that their management isn't taking it seriously enough. Is that a PR stunt too?
They "called it a day" when they decided to deploy an agent which they believe to be dangerous inside this poorly secured environment.
There was no real isolation because a part of the system that doesn't provide any isolation guarantees was bridged to the internet. The next version of Artifactory, which you definitely wouldn't audit before you rolled it out, could simply add a public API that sends requests out to the internet.
Such an innocent upstream change would be equally catastrophic for your security model, which should demonstrate why it's negligent to rely on undefined behavior to enforce your security policies.
This should frankly be obvious any operator entrusted with running dangerous and possibly malicious code. Even if you don't know what you're doing, any LLM would tell you that this is a really bad idea if you simply asked. Don't rely on spacebar heating [1] to keep humanity alive.
[1] https://xkcd.com/1172/
If it is "extremely powerful" then my concern is not AI going rogue but a concentration of power of the few companies that decide how it is used and distributed not websites getting hacked.
So powerful they can't even do math properly without a staff worth millions writing all the clutches so that they can, wooooowooooooo
> OpenAI, for example, thought their sandboxes were good enough. As their AIs got more and more advanced, they kept proving them wrong - sandbox after sandbox.
Were their sandboxes in-process with the harness? Were they actually better than something like bubblewrap or even docker?
You mean the sandbox that wasn't airgapped from the public internet? The one that wasn't even a separate VM? The one that was barely a chroot? That sandbox?
>Because AI genuinely is an extremely powerful and extremely dangerous
I urge you to consider the elephant in the room you failed to mention if you earnestly believe this then.
In what world is anyone allowed to sell something they expressedly know to be "extremely dangerous" to the public?
Let's say I created a lethal pathogen that I know to be lethal in certain common environments but also know it acts as a "no side-effects" antidepressent for people in certain other environments and I release it knowing full well I can't control it.
When people start dying can I defend myself by saying, "Well I said and documented that it was extremely dangerous and no one came to stop me, so I don't see how you can blame me...If I didn't do it someone else would have."
this happens all the time. have you heard of the Sackler family and the opioid crisis?
the difference here is that they’re calling for someone to stop them, which is both weird and unconvincing because these immensely powerful billionaires can in fact make their own decisions
>the difference here is that they’re calling for someone to stop them
Jesus Christ, that's my entire point. it's not remotely the same thing since the Sackler family wasn't going around telling people opoids were extremely dangerous while pushing them on everyone.
The point is if they know something to be extremely dangerous and keep doing it people would expect them to get prosecuted.
Is it a hallucination machine and a stochastic parrot or is it so powerful we need it to be controlled like nuclear weaponry
If I have a random number generator and I also have the burning desire to connect the random number generator to the nuclear launch system it is both just a random number generator and also I should be tackled to the ground by serious men with sunglasses in black suits and sent to prison.
a hallucination machine on an infinite loop and infinite tokens is a dangerous combination.
We need adversarial peer review. All corporations and projects should have ai safety tests that applied equally, with results and tests that everyone can inspect and challenge and improve.
Just read the tone of this article:
> said Dr. Andrew Lenson, a Senior Lecturer in all kinds of science sounding stuff at Victoria University.
Don't we need more humorous yet serious writing like this in the world.
Why aren't the AI companies hacking each other?
Then we can truly see who is more dangerous?
:)
And I'm waiting for the story of Amodei sending a 6ft humanoid robot to OpenAI headquarters asking for Sam Altman.
Of course it must be dangerous - how else would you pitch it to the military?
We are living in a timeline written by The Onion.
Here's a relevant article by the Onion from the summer of 2022, before ChatGPT. https://theonion.com/ceos-warn-against-the-dangers-of-artifi...
Well, or at least by The Civilian, a New Zealand version of The Onion.
Anything to take attention off the fact that these companies are burnings piles of cash and questionable future spend commitments with no sign of that stopping any time soon.
And lo and behold, nobody even remembers their promises to IPO this year. Can't now, it's too dangerous!
A recent Saturday Night Live clip featuring an impression of Amodei that hits on this theme,
https://www.youtube.com/watch?v=-Nvne3LzBls
I hated the FUD when Microsoft were funnelling millions into it. Especially as I really liked their frameworks, languages and some of their operating systems and tools (Windows 2000, Windows XP, Windows 7, SQL Server, .Net, Office 2000, Office 2003 and Office 2007 were all excellent products).
Now the two AI companies born with help of Microsoft have a decent core product but have gone full FUD slop on us.
If I were the US government I would force the AI labs to solve the jobs that nobody likes to do, like cleaning the streets and collecting garbage. Because those are the jobs we will be doing if developments continue as they have been.
You won't be doing any mandatory jobs, you'll be living in a post scarcity economy.
So far I've only seen AI taking the fun jobs.
I haven't seen AI taking any jobs yet, maybe I'm not looking hard enough.
Stop posting garbage.
> Australian Prime Minister Anthony Albanese spoke with Altman to express “extreme concern” about the incident, a compliment Altman said he greatly appreciated.
"a compliment Altman said he greatly appreciated" is fabricated; there is no source for this.
Its like the onion.
it’s satire…
Can they demonstrate that this crap actually makes money?
Does it need to? They only need enough funding to make it through to building superintelligence. That's probably 3-5 years out, and they will very easily be able to get enough funding to last that long.
lol
That's why they have their LLMs attack everybody. They will be way too busy repairing their hacked web sites to ask that question. Also, once humanity is successfully destroyed, money will be obsolete anyway.
If this approach works with AI companies, why the same doesn't work for nuclear power plants? :)
Except that their employees are quitting 7-figure jobs because they believe what they're building is threatening to humanity.
Maybe take these people at face value.
On the other hand, they've received a 7-figure salary. Some people would do a year of that and then fuck off to the beach with their family and not work another day in their life, while throwing prognostications out the window as they leave.
The problem is people who see the lavish pay and then reach a goal and then quit and retire. The interview loop didn't cover that. And why should it?
this is good for them because they can call for regulation and since they have infinite money, they can survive regulation and hurt labs that do not have infinite funding / open weights ones
I wish they would call their bluff. Just declare them a national security concern and send in the feds. Lets see how fast they back peddle on this. Any if their models are hacking everything like they claim they should be charged for it like any normie would be if they did the same.
I suspect that TLAs and the administration are already too hooked to make that move. One can wish. It would be the legal thing to do.
At this point, AI companies are basically competing over who can destroy humanity first. What a time to be alive.
I remember when HN scoffed at the idea of Mythos hacking anything.
Imagine if the atom bomb was created by private companies about to make their IPO.
Honestly, I'm running out of ways to express what Sam and Dario are. I'm now down to 1 word: Wankers.
I had to double check if this was an Onion article or not.
Oh dear. I appear to still be stuck in the onionverse.
My opinion is that they want to score military contracts. Once private money dries up because AI doesn't deliver 10x productivity, they can still convince the society to invest in AI because if the society doesn't, the enemy will invest in bigger AI.
Case in point - atomic bombs. France paid through the nose to have own nukes. US and USSR had a very expensive dick measuring contest about it.
Somehow, not the Onion.
Manipulation through fear
They need to control, regulate and tax AI so the first line of attack is the sheeple, so tell them the water is going to disappear and everybody will die a slow death of anguish, thirst and drought like never seen. Then evoke terminator apocalypses where drones go rogue and kill half the world. Then send their most likable emissaries with puppy eyes asking for compassion about our children's future
Then they ask for your vote (oh democracy)
In the end all they want is control and money, while keeping their competitors (china) in check, backed by the outrage of the weak majority
Politics as usual, move along
I swear so many people on HN are developing psychosis.
The much simpler solution is that they are building superintelligence, which they think can be used for great good (like curing most death and disease), but it is also a powerful capability that obviously needs to be regulated and is at least somewhat a matter of national security.
But nooo! Surely they are building.... terminator bots!? JFC.
> like curing most death and disease
Because people are giving them benefit of the doubt. They assume it is about something sane rather then "the goal is for some remaining humans to merge with tech as immortals and fly to the stars".
It's just marketing. They want people to believe they're selling super intelligence that is powerful enough to destroy the world of it fell into the wrong hands. It's much more effective than saying that you have a slightly more effective slop generator which obviously has no actual intelligence.
Even the models we have access to can do basic coding tasks really well, in a way they couldn't a year ago.
I don't really care what you call "intelligence" - what matters is capability, and the consequences of that capability.
They're not selling the superintelligence yet - the models in the Hacking Face incident and that solved the Millennium prize are not released yet. And that is by no means the final capability level the current process looks like it will get to.
Meanwhile, users don't seem to care one iota.
It amuses me to think that there could a noble aim: they're overdoing it partly to raise awareness of the real dangers. It's something we get as a side effect of their desire to achieve regulatory capture.
FWIW, the site's "quote of the week" looks entirely nonsensical and made up.
Why is this written in the same tone as an article on The Onion? Is this pure satire?
Yes… It is clearly satire. It’s funny to see how many people don’t realize it…
It's especially funny because they are the people that the article is making fun of.
The Civilian is the NZ version of The Onion.
This is the funniest timeline. Companies begging for heavy handed regulations because they promise their products will destroy all life on Earth. Governments absolutely refusing to govern. CEO of world's most valuable chip company interviewing in his Fonzie leather jacket telling people to BUY MORE OF MY PRODUCT BECAUSE THOSE SCIENTISTS DON'T KNOW SHIT ABOUT SHIT! World's richest capitalist promising we will have communism in 10 years.
If people had known about a civilian, peacetime Manhattan Project and were in a modern media environment where we could kibitz about it, would it have been slowed down?
Is AI/AIG too powerful to be undertaken by entities that are not beholden to our constitution? (I know that doesn't seem like much in the Trump era, but I hope for a return to government norms relatively soon.)
pretend, not demonstrate
"These models are harmless and it's all just marketing" is the HN equivalent of Covid-truthing. I was responding to someone on Bluesky recently who claimed the METR report said the whole coordination thing was hallucinated by agents reading the logs. Hadn't read it themselves of course, and were taking tiny fragments out of context to support it. These people seem to either believe that there's not really been any malicious agent access, or that all systems that can wreak havoc on us are perfectly air-gapped. I really can't wrap my head around it.
Not that it's harmless, is just a point being forced into the public discourse to force an outcome, which in this case is not a ban, just a buy in from the government for regulatory capture.
If my neighbor owned a kennel, removed all the fences, and his violent dogs attacked five children in the past month, he would be in jail rather than bragging about how his dogs somehow gained autonomous behaviour.
What if he predicted that dog technology would eventually lead to dogs capable of destroying every fence in the world and wanted to make sure people aware of this problem would be the ones who did exactly that. Than speed-ran experiments on dogs which made them very attractive for economic purposes but also started to get very capable at destroying fences as predicted. We should work together or with lawsuits if necessary to keep the dogs within their fences. But we should also start wondering about dogs, economics and fences because what we know about them is about to radically change.
You're forgetting that those fences were made of paper.
If you connect your fuzzer to nuclear weapons, of course it'll be extremely dangerous. But that's obviously your fault to make the connection.
As I understand it, you're arguing that all sufficiently dangerous systems are well enough hardened or air-gapped, and that people don't exist who wish to cause harm using these systems? Did I understand right?
To launch a nuclear missile you need to physically insert a launch code than only the president has closed in a suitcase that is well guarded, as well as physically turn two keys that are enough distant apart that one person alone couldn't.
Now, unless your AI gains access to the codes, and sends two robots in a bunker or nuclear submarine to insert the code and turn the keys to launch the missile, you can be modestly sure that no AI could launch a nuclear missile or nonsense like that.
I'm quite frankly more worried that some foolish president decides one day to launch a nuclear missile than an AI could do it.
Now that we return to the real world, the only thing AI could do is to attach systems that are connected to the internet (and critical systems are not), but let's be real, AI are not really that intelligent by themself, it's not that someday an AI could decide like in Matrix or Terminator to exterminate humans, because (at least now) AI have no consciousness and can't really decide what to do.
An human can of course use an AI to carry out cyberattacks, as he can do that also without AI like it's done since the internet exists, but it's an entirely different thing (a human using a tool, AI, to do damage VS the AI itself that decides by himself to destroy the human race).
It's public knowledge that the code is 000000 because the risk of the president being incapacitated making it impossible to set the bombs off was judged worse than the risk of accidentally setting them off.
If James Watt had gone to the government and public saying how dangerous the steam engine was, and how many people would perish in boiler explosions if he wasn't given exclusive control over all steam engines henceforth, he would have been rightly called a swindler and laughed out of the room. He would have been correct that they can be dangerous, but utterly wrong to suggest that warranted him having a personal monopoly or oligopoly.
The truthers will be the first one to say I told you so when agents do another dangerous thing autonomously.
For truthers AI is simultaneously a machine that hallucinates 100% of the time but also so powerful that it can cause nuclear levels of destruction. Because something something “wrong hands”.
Seriously:
Domyn's CEO says OpenAI, Anthropic are lying about safety
https://news.ycombinator.com/item?id=49875725
If true we should especially sue OpenAI for hacking, and will find this conspiracy in discovery.
Or, find their scaling evidence, and corner cutting on safety.
So hope you agree and are pushing for that!
Can't wait. They've been criminally negligent in most jurisdictions. Trump can help you outside the US.
I completely agree, but:
List of fraudsters pardoned by Trump
https://www.businessinsider.com/list-billionaires-businesspe...
The current attorney general and former Trump lawyer on Fox News yesterday on AI (sorry, "super intelligence"):
https://bsky.app/profile/atrupar.com/post/3mwissj34rt22
Anthropic's Dario Amodei to have White House dinner with Trump on Sunday:
https://www.axios.com/2026/09/27/anthropic-trump-dario-amode...
77 million Americans brought this onto themselves, and all of us.
Consider this "strong" premise:
Consider also this "weaker" form of the same argument: Here's a few hard truths that no one will like to hear:1. Those who accept and advocate for either or both of these premises do so based on ideology, and not based on the scientific method.
This is not a controversial statement. Advocates of the "AI danger" premise (whether the strong or weak forms above) freely and gladly, even insistently, note that the conclusive proof of what they're describing has never been observed, much less observed enough for any kind of experimentation loop to have run for any meaningful number of iterations. They are generally fine with accepting the premises without conclusive proof, because, they theorize, the instant that a conclusive proof event occurs and we observe it for the first time, humanity goes extinct shortly after. So if either premise is true, the consequences of them being true can only be averted by accepting the possibility / probability / certainty of them being true and acting to prevent their consequences without waiting for conclusive proof.
2. Statement 1, which asserts a brute fact and not my or anyone else's opinion, is not a commentary on whether either premise is in fact true or not. It is only a commentary on the epistemics of those who broadly accept them.
3. Since the conclusive proof is not available, advocates of the "AI danger" premise argue publicly for their view based on what they acknowledge (again, freely and gladly) to be inconclusive proofs.
Popular in this set of inconclusive proofs is the Hugging Face hack, which allegedly demonstrates that so-branded "AI" technologies are capable of becoming inherently or autonomously dangerous at a scale that threatens humanity to either the greater or lesser degree above (this is a different assertion than the one that claims they're already that inherently or autonomously dangerous today).
4. While we have plenty of facts about these inconclusive proof events, the facts most critical to their interpretation as supporting the premises above come from biased/non-neutral sources.
The chain of events that transpired in the Hugging Face attack are publicly documented perhaps better than any other cybersecurity event in history. But the details that make this event as a whole either compelling or not compelling in support of the "AI danger" premise (either strong or weak form) come from within OpenAI itself, and are prone just as much to selective omission as direct manipulation. These are details like:
- What exactly were the models that exhibited this behavior, and what were they pre- and post-trained on?
- Were the agents pushed at all by human intention toward creating a public spectacle, the way they subsequently did?
- Why were so many agents run on this task in parallel and what did OpenAI expect the return on investment of so much compute world be?
- Was the weak sandboxing known about prior to the attack? Was it purposefully either arranged that way or noticed and not fixed?
Reliable answers to any of these questions world completely change the salience of the Hugging Face attack to the "AI danger" premise, and on every one of them we have to take OpenAI's word for it (or else be content when they choose not to address it one way or the other).
So in brief: we don't know anything for sure, what we'd like to think we know comes from unreliable sources, the loudest voices advocating for the most sweeping change are ideologically driven, and everybody with access to ground truth has maximal incentive to blur, bend, or break the conveyance of that truth to the public. So what do we even expect for our own ability to discern reality from fiction on this topic? We should expect little, if any ability at all.
I do not need a model to know that selfish mega-corporations cause a lot of harm.
Can we get our money back here? I mean, just take the increase in RAM prices alone. That should come with a huge penalty they should give us back due to collusion. And that's just one of many more things. Trump's family dynasty also owes us all money - we want that back as well.
Meanwhile Chinese companies just use massive government subsidy to distill American models and give the models to everyone for free. That's true communism. /s
Yeah totally unlike the US where tech companies are awarded multi-billion dollar contracts of taxpayer money in no-bid contracts without oversight
And where the Vice President has almost exclusively only been employed for the primary investor in several such companies
Pretty sure you can check where Chinese government subsidies goes, and AI companies are not the target. Are they benefitting from Chinese government subsidies to powerplants and semiconductors? Yes, of course, but they don't benefit from direct help.
Meanwhile US models just use massive investor subsidy to distill American copyrighted work and rent the models to everyone for money. That's true capitalism. /!s
> Chinese companies just use massive government subsidy
[citation needed]
> Where companies previously competed to demonstrate superior abilities in programming, passive-aggressive emails, and videos of unusually high slides, they’re now seeking to woo customers with the claim that their model is the one currently most capable of ending the human race.
You know we’re living it off times when this is not an Onion article.
> And even more dramatically, when asked about rumours that its model, Claude, had killed his wife by hacking into the family’s WiFi-connected microwave, Anthropic CEO Dario Amodei replied “Well, yeah, sometimes.”
I had to read that twice. "Oh okay. The whole thing was satire."
When the Onion is not the only website that publishes satire, what’s that a sign of?
It is a parody site.
> OpenAI CEO Sam Altman celebrated the breach as an “alarming threat to cybersecurity.”
This is a satire website too.
"said Dr. Andrew Lenson, a Senior Lecturer in all kinds of science sounding stuff"
"Anthropic shares skyrocketed upon the revelations; a remarkable development, particularly given the company is privately owned."
Perhaps it's mocking the misleading press coverage of AI companies?