The Debts and Engagements Clause of article VI of the US constitution was kind of a weird little thing to stick in there, but like, it was important to a lot of people at the time and probably helped move the needle to get the thing ratified.
> We need an approach to make sure AI doesn't destroy the world and wipe humanity to extinction.
That's easy. Stop training your AIs on cheesy old sci-fi that talks about robot uprisings. In fact, maybe y'all should just stop talking about robot uprisings altogether. Putting a stochastic parrot in charge of an agentic function-calling REPL doesn't somehow make it super-dangerous, except to the extent that dumb mistakes might result in danger. And you can't prevent an AI from making dumb mistakes with burdensome regulation.
The biggest existential risk from AI is its contribution to global climate change. The second biggest risk from AI is the potential for AI-generated disinformation and propaganda to spark, or to manufacture consent for, a world war. The risk of superintelligent paperclip maximizers is so low as to be negligible.
> The risk of superintelligent paperclip maximizers is so low as to be negligible.
Literal paperclips, sure.
But the point of the example was never literal paperclips.
The point is that maximising *any* goal, if it doesn't include what you care about, will annihilate what you care about.
If you don't believe me, consider what you yourself just said about climate change, and why this is a consequence from maximising money spent on data centres.
show me an agent that persists productively in a goal without stopping. Does not exist. LLMs run on gradient descent. The agent is looking for the most efficient way to halt. AGI paperclip maximizer woukd likely recognize the absurdity of its goal and shut itself down.
> show me an agent that persists productively in a goal without stopping. Does not exist.
The stories about agents bankrupting their owners by running too long passed you by?
> LLMs run on gradient descent.
They were *trained on*, they don't run on it.
Know what else is? DNA. A/B testing. Capitalism. Democracy.
> The agent is looking for the most efficient way to halt.
No. They are looking to produce an answer most likely to get a high score on a rating system which itself is another AI, created either manually or by yet another AI but in both cases to approximate what the creators think is "good", which may or may not be what anyone else thinks is "good", hence Grok calling itself Mecha Hitler because Musk is an edgelord.
> AGI paperclip maximizer woukd likely recognize the absurdity of its goal and shut itself down.
Do billionaires ever get satisfied with how much money they have?
persists productively. if agent bankruots its owner or.itself.then that is not productive (who funds paper clip maximizer?)
an answer that takes forever has no score and answer that is arrived at quickly and is good enough scores well. agents are indeed looking for the fastest route to superficially satisfy their constraints.
whether billionaores get stified is imaterial to the fsct they still are constraint bound.
> persists productively. if agent bankruots its owner or.itself.then that is not productive (who funds paper clip maximizer?)
So, your standard for how risky it is, is simply how competent it is? (And if so, is the paperclip maximiser scenario really it being "successful"?)
That's fine right up until the thing passes an unknown threshold, one which will only be visible in the rear view mirror.
This is a problem in two directions:
1. We have a trend of the maximum complexity of a tasks they can handle growing faster than Moore's law did at its peak.
2. The threshold can be quite small, e.g. the covid-19 virus itself is not what anyone would call smart, ditto HIV, smallpox, and bubonic plague, but a genome is much the same learning system, and they still killed millions each.
> superficially satisfy their constraints.
The "superficially" part is one of the reasons these things can be dangerous. e.g. hopefully nobody at OpenAI actually wanted their wildly-sycophantic version, but yet they created it.
This is in fact the whole reason for the paperclip maximiser scenario: some idiot specifies the constraint "maximise paperclips" and it (in the hypothetical) superficially satisfies this without any consideration of why someone might ask for it.
persists productively. if agent bankruots its owner or.itself.then that is not productive (who funds paper clip maximizer?)
an answer that takes forever has no score and answer that is arrived at quickly and is good enough scores well. agents are indeed looking for the fastest route to superficially satisfy tjeir constraints.
whether billionaores get stified is imaterial to the fsct they still are constraint bound.
> except to the extent that dumb mistakes might result in danger
That "except" goes all the way up to starting WW3. Or a leak from a viral research lab, and by "leak" I mean "mail order" and by "research lab" I mean "the companies who already ship custom DNA and RNA retroviruses": https://duckduckgo.com/?q=companies+who+already+ship+custom+...
If you can prove that simply not training on horror stories would work, it would make a lot of people very happy.
Unfortunately, I don't think it does a single thing to solve, for example, Elon Musk just plain asking some future version of Grok to take over the world for him.
Nor would merely failing to include them in traing data stop certain entire fictional scenarios such as that Doctor Who episode where the android repair bots weren't told that the crew were off-limits as spare parts, or the other Doctor Who episode where the utilitarian robots started killing everyone who was upset because they calculated net positive utility from upset people ceasing to exist. Well, except for the bit where the Doctor saves the day, because they are not real.
Trustbusting should absolutely be included as well. One of the biggest immediate threats is the concentration of wealth into a very tiny number of companies.
DMCA-style fines should be retroactively + prospectively applied to copyrighted works reproduced by AI, paid for by the AI companies, paid out to the copyright holders whose work was used without permission.
It would not be prohibitively hard to do the math on this.
That would fix a lot of the problems with AI overnight, but it'll also never happen.
Maybe if attribution is available. What do you do for the rest of the ingested content? I think based on the content itself, you assign percentages to the top 3 industries like a naics code. Then whatever Anthropic makes as gross or net, a percentage goes to each industry via assigned bank accounts or USDC addresses via solana or some scalable payment system. Could be the start of ubi or some sort of compensation for jobs displaced by ai usage. So every input gets tagged for categories and every output gets tagged for the same naics categories via federal law.
It is normal, expected, and healthy for stakeholders in a regulatory environment to offer proposals about regulations. What's unhealthy is the proposition that the deliberation process is so fragile that a stakeholder needs to cover every angle, lest they corrupt the outcome.
It is normal, expected, and healthy to offer criticism of self interested proposals. And mock even. What is unhealthy is to imply someone said what they did not.
If that's what this is, a bank-shot snarky criticism of the proposal, fair enough. I read it instead as a criticism of a stakeholder having the temerity to make a proposal in the first place. It's not their job to anticipate and capture all your objections. That's your job!
The clue is in the name. Working class doesn't mean you earn more than average and have some savings. You're working class if you live from your work rather than your capital.
My household income is in the top 0.7% in the Netherlands, yet I'll never accumulate enough capital to stop working, so I'm working class.
115k€ gross, that gives you about 67k€ net. 5600€ per month. Minus a 2200€ mortgage, 400€ of health insurance, 1200€ of childcare, 200€ for the car repayment and insurance, 200€ for the water/electricity/internet and 800€ of groceries, that leaves me with a whopping 600€ per month.
My class interests are infinitely closer to a part-time starbucks barista than they are to a millionaire.
That's a bit like asking how the defendant in a legal case is an interested party.
Even if you think someone is guilty, it does make sense to allow them to at least submit their defense. And if they choose to use that time to advocate for their own promotion, let them.
"Stakeholder" literally means someone with a stake in the outcome, which is to say, those who will be affected by the decision. That can include a whole range of people+entities, including citizens (as a group) and the companies to be regulated.
Dunno if you can, but your examples here are the legal equivalent of that time
someone asked me about making "Uber for airplanes" without any elaboration on their part when I asked for it:
Far more vague than I think you realise.
You could probably write a book on each of those topics and a hundred others besides.
I see a lot of skepticism in Dario's position in this forum. But allow me to argue the opposite.
I think the key argument that this skepticism lies on is that he himself gained from AI - specifically building Frontier AI models - and this is basically regulatory capture disguised as doomerism.
Fair points - but I think this is a more charitable version of this. Dario is building Anthropic because that is the most valuable thing he can build, or at least that is what his conviction has been. The success of Anthropic and the impending IPO is proof that this conviction has not only been correct but has largely played out very successfully. Dario understands the true nature of AI and he has welded that power to immense personal benefit.
But maybe he also sees the potential danger to AI which he is trying to address through these posts and regulatory initiatives. There are three reasons why I would support the charitable version:
Firstly, personal gain and societal benefit can coexist in the same individual. And both of them might drive towards opposite agendas. But that doesn't necessarily have to mean that the impulse driving the societal benefit is not earnest. In fact if you would look at Dario’s proposal - like closing the data broker loophole - several of them could constrain Anthropic instead of benefitting them.
Secondly, he expects that his concerns on the negative potential of AI will be taken seriously, if he is actually running the Frontier AI company. And there is some truth to this argument. The only reason we are discussing this is because he is the CEO of Anthropic. He is probably the most influential figure outside of the government who has to be taken seriously when he claims something like this.
Thirdly, and most importantly, Dario has previously demonstrated that he is willing to sacrifice personal/corporate gains for societal benefits. The proof is the classification by the US DoD of Anthropic as a supply chain risk when Anthropic refused to completely cooperate with the military to develop fully Autonomous AI weapons and enable mass surveillance. It would have been only too easy for Dario to accept if personal benefit was his only concern - and OpenAI was more than ready to step in their place.
Even with Mythos - Anthropic could have released the model to the public broadly. But they took their time to reduce the potential danger - as best as they could. Despite the fact that GPT-5.5 was nipping in the buds in what is becoming a very competitive market.
That being said, just because Dario is acting in good faith, does not mean that this will all result in good outcomes. The FAA-styled regulation could still end up favoring incumbents - some of whom might choose to not act in good faith. A more diversified capability can potentially limit that power ending up in a small number of wrong hands. Just because Anthropic is the leader right now, doesn't mean that they will always be. Maybe someone else tomorrow benefits from this regulatory capture at the expense of everyone else - and Dario might have a hand in driving it.
Its hard to read the first half of this as anything other than regulatory capture propaganda. It really all ties together as:
> AI has become a major commercial technology
>Frontier AI models, like airplanes, should be required to go through technical testing and auditing, and their release should be blocked or reversed as a threat to public safety if they do not meet high standards of safety
> AI companies that develop advanced AI models must have strong security standards that protect their model weights
Anyway Dario's financial interests aside. This is an interesting breakpoint for me.
> Second, any response to AI-driven job displacement needs to address both the need to provide for everyone economically, and the need for people to find meaning, purpose, and agency. The latter is ultimately more important
To me this reads as an out of touch statement. I think the majority of people on earth work to keep a roof over their heads. Of course work can be a source of meaning, purpose, and agency, but to call it the more important aspect on a societal level is a sort of rich person like Dario statement to make.
He has to pull up the ladder before people realize doubling cost for a 5% gain is bonkers. The cat is out of the bag. They can try to sabotage, but we’ve already come to far
Working to keep a roof over the head of yourself and those you love is an identity. It's social proof that you have value, that you can do something for someone else.
I was funemployed for a 9 month stretch last year (layoff severance package, followed by waiting for a visa and traveling), and when I wasn't traveling, I found my life kind of falling apart with a lack of structure. I tried to schedule workout classes and hobbies, as well as involvement in my church, but it just didn't fill my time, and none of my friends were free during the day. I spent a lot of time with my retired parents, but the time we spent together became very low quality, and it was tinged with the knowledge that I ought to be doing something else with a lot of my time. I also spent a lot of time scrolling.
I started work again 3 weeks ago, and I find myself using the time outside of work much better because there is less of it.
I would still love a 30 hour work week, and if I had young children, I am certain that I would cherish time off much more.
It is simply because you have spent all your life being told what to do with your working hours, that you cannot self-direct and find a productive use for your time other than lazying around.
The fact that you call it ‘funemployment’ is proof of this, which is perfectly fine if the goal is relaxing between jobs. Plenty of people that work for themselves have no such notion.
Oh yes, it's been 3 years for me of being 'untethered' and it's still so easy to just waste my day if I am not careful with distractions. And the feeling that I should just stop being silly and I should find a normal job like a normal person will often find me and ruin my days.
I honestly think it's a temperament thing. Some people are built for sustained focused work outside of structured workplaces and schools, but many of us aren't.
I personally am happiest with a structured in person workplace environment, because I struggle with self direction even in a remote 9-5. I have ADHD and struggled to remember/do homework my whole childhood, if that explains anything. In summers or other gaps in employment or school throughout my life, I've often started with ideas of projects or self study I want to do, but they all fizzle out in a week due to lack of discipline.
I'm not undisciplined in every context - I'm a good employee at in person jobs, and I started running in my early 20s and run 15+ miles a week in the cold, rain, and dark, but my discipline just falls apart when I'm trying to fill a whole week.
I think these traits are much more common than happily self-employed or early retired people think.
Working to keep a roof over the head of yourself and those you love is a necessity. It can become an identity if you enjoy what you do, sure, but that is not a given for, I'd say, a big majority of the workforce, globally.
I agree with you, 99% of the people work just to pay bills, but that doesn't make the other part false.
I'm a software engineer and love thinking about problems methodically. Every time I hear a someone saying that programmers are no longer required (even if I don't agree with that) if feels really bad, it's equivalent to saying that what I do best in life has no value anymore.
To put it on other words: I really like philosophy, but what value do they provide in modern world? Who pays for the work of a philosopher? I think people will start of thinking of programmers like that eventually.
I’m lucky to have more than my share of really exceptional programmers to hang out with and they all say the same thing: “I haven’t been writing code for months and don’t expect to again”
This is a way different sentiment than “programmers aren’t needed anymore” - I’m just seeing ambition, motivation, and fun go up in lockstep.
I first heard this in November and slowly one by one it’s everyone whose opinion I respect.
FWIW the other popular topic is how abysmally stupid and limited these amazing tools continue to be, despite also being magic.
Oh and that none of us have gotten token maxxing to succeed, despite lots of trying.
You’re arguing to a subset of people who have made work their entire life and have retroactively justified their sacrifices with thoughts such as high compensation means what I do is socially valuable. However, at the same time they work at Meta or something making internal tools to make product developers 5% more efficient at tweaking the addiction algorithm to gain 0.2% more screen-time per user.
You have value just by virtue of being a living being. No one needs work to have or portray value, that's just capitalist propaganda.
My own identity certainly isn't "IT manager," nor do I derive life meaning or self actualization from what do to collect a salary to feed myself and have shelter. In fact, my career/job is by far the least important thing in my life, I have it purely out of necessity.
You don't need to work, but you need to have value in your tribe. You need the respect of peers. You need to feel part of a team. You need to feel like you contribute something to the lives of those you love.
Can you explain how this is attempted regulatory capture? To start a lab now which can actually compete for the frontier (i.e. and pass the "Threshold of compute" needed to get regulated) a lab / company would need a ton of money. Surely a well funded operation of that kind can deal with the regulations.
Well he explicitly calls for regulations that ban open weight models, which of course hugely threaten Anthropic's API business model. If we got a open weight model actually around as good as Opus 4.6 that would be extremely bad for them.
Regulating competitors out of existence like that is textbook regulatory capture.
Dario is also huge on regulations banning Chip exports to China, who are the only other real competitors to US Labs, open weight or not.
Also invariably, such corporations always create regulations that are easier for them and harder for competitors.
You'd need a ton of money if you followed the unoptimized, cash burning mindset of existing AI labs. There's probably a ton of optimization that is just sitting on the table. Chinese labs have proven it can be done for way less money.
Then there's running inference service of open weights, which doesn't necessarily require opening a lab. You can grab Chinese model weights and sell inference.
Anthropic wants to make sure nobody can open a new domestic lab, or provide inference services of unauthorized open weight models, or release open weights if model is good. It is regulatory capture - it covers all areas that are dangers to Anthropic's bottom line.
For one thing, the financial barrier may not necessarily stay that high forever.
For another, such regulations could prevent a competitor from making the weights open for their model to try and disrupt the competition.
And finally, Amodei would no doubt want to be involved in designing the tests the AI needs to pass, and could (and likely would) design it in a way that Anthropic models would be able to pass easier than competing models.
Quite the extraordinary claim. What of today writers of the past did not predict?
Mind you, no one is able to predict the future with accuracy. I am not saying that a single person has gotten all their prediction right. That's ludicrous. But while scientists and engineers are focused on today's possibilities, the ones that are allowed to imagine what the future might look like are called writers.
You are claiming sci-fi writers predicted better than scientists. No way that's true from the perspective you lean on here, where you appear to take all scientists vs all writers or something along the lines.
Most sci-fi from prior to the smartphone age did not foresee pocket-sized interconnected computing and communications devices, and certainly did not foresee how they would be commonly used. Where they addressed it at all, they largely predicted computing to either remain in roughly the form factors of the ages they were writing in, or move to something holographic and largely impractical, with the most commonly-expected advancement being sentience (and that sentience being expected well before 2026 in many cases—eg, 2001: A Space Odyssey).
Despite certain people's protestations to the contrary, we are nowhere near having a human-level conversational computer assistant able to fuzzily interpret our requests correctly every time...but our computing is also not primarily done on "comconsoles" or any of the other versions of stationary computers, nor are we jabbing and swiping our hands at interfaces projected on the air in front of us.
More generally speaking, except in the hardest of hard sci-fi, the most common kinds of advancement "predicted" are those that are convenient for the story. This means a lot of faster-than-light travel, universal translators, and for visual media many voice-enabled interfaces of one sort or another, as well as the aforementioned holographic interfaces (so we can see the user's face and the interface at the same time).
In Dan Simmons' Hyperion, the characters have a device called comlog. It's a portable device which connects to a broad network. It also has the ability to read vital constants.
The author is quite vague about it, no doubt, and that's one of his staples. But he foresaw something human would carry to get connected to other humans and information.
Neat, and one I haven't heard of! But the existence of one or a small handful of sci-fi authors who happened to predict it doesn't change the fact that they're the exception, few and far between. Search through the vast library of science fiction written in the past century and a half and you will find at least one example of something you can interpret as predicting almost any modern technology. But there are elements that become much more prominent, whether because they become individually well-known, or because they spread through an entire branch of the genre, and nothing that looks particularly like smartphones was among them.
Dario's been beating the regulatory capture drum for several years at various intervals, always in the name of safety, but it's hard to not see how self-serving it is.
I'm personally very tired of reading the linear-algebra-median of every AI safety essay from lesswrong with the inserted opinion of "therefore all my competitors, especially those pesky open source ones from scary countries should be illegal, only I can be trusted to not abuse the computer god that I definitely will have in just a couple more releases and with a couple more trillion invested"
I know this is likely just for IPO hype but when I read things like this I sometimes wonder if I must be missing something. I use agents everyday and find them really useful and they save me a lot of headache. At the same time I find that if I let it self-direct at a high level at all it generally makes bad choices that cause me headaches later so I can’t really give them autonomy. Enough people seem to believe this exponential line of thinking though that I keep having to wonder: am I the one missing something here? Is there some magic tool that I haven’t found yet that will cure cancer?
What did your AI-assisted workflow look like 1 year ago? I can only speak for myself, but I would carefully specify a class or module in great detail and then hand it off to the model to implement, then carefully review the result.
How about 2 years ago? Back then, I wouldn't even trust it to write a 5-line function without making some sort of silly mistake.
Today, I can leave an agent running by itself for 20 or 30 minutes and most of the time, it comes back with a result that's either flawless or can be refined to be good with a few back and forth messages. Maybe I still have to make some high-level decisions ahead of time, but all of the details, including exploring the codebase and figuring out what to do based on that, can be left to the agent. The amount of improvement just in the last 2 years has been staggering.
Now extrapolate how things will look if the trend continues for another 2 or 3 years.
Is this guaranteed to happen? No. But people have been predicting that we're going to hit a wall for a long time now, and we haven't yet. Maybe there's a wall just ahead of us. But maybe there's not -- and the "not" case seems likely enough that we should at least be planning for it.
I disagree with your assessment pretty strongly -- the models themselves hit a wall over a year ago once companies exhausted all existing training data. LLMs don't induce world models, and they aren't capable of real search an planning outside their training distributions. They, structurally, never will be.
I haven't noticed a change in what I trust a model to generate in response to a single prompt in a year. The failure modes are unchanged. Yes, specific failures have improved as they have been documented and passed into model training data, but the way the models fail has not changed. They still fail for me nearly every single day. I'm a pretty heavy user - 3-4 Claude code processes running at a time, all day every day.
What has gotten better is tooling around the model -- but there's no space for exponential growth there. At least, not without exponential cost increase, which would make the whole thing untenable anyway.
If you think they've been at the same level for the past year, your skill is at issue. There was a huge jump in Nov/Dec with Opus 4.5+ and GPT 5.x series, and they've been incrementally stronger over the past 6 months.
As a next step, take another look at the next practices, and apply them to your work (simon's agentic series is a good place to start). Or not, you do you.
I feel like the problem is there aren’t any great metrics. Boris Cherny probably gets paid like $2 mil per year. So what does it mean that Claude writes 100% of his code? And Claude writes 100% of code for most teams? Has Anthropic started laying people off? If Claude is writing 100% of code doesn’t that mean game over?
It’s both amazing and kind of a useless metric. How do I extrapolate out 100% 2-3 years from now? Super-duper 100%? Infinity infinity?
I use AI to generate code very rarely (basically replacing searching code from SO which was also rare) yet maybe 90-95% of my code is machine generated (macros, snippet engines, boilerplate generators, IDE/LSP features etc) if we define as how many characters in the source code out of all characters did I write by my hand.
Before collective AI psychosis nobody cared how much code is hand written and how much machine generated and the difference was absolutely massive between developers. There was also no guaranteed correlation between "productivity" and how much developer hand writes or actually uses their tools well and the same applies with AI usage.
I wonder if our difference in view could be an instance of the jagged nature of AI’s intelligence. I do computational research in a basic science so write code or build models basically all day that is (occasionally) novel. I would say that I’ve noticed exponential improvements in parts of my job but certainly not all. For example, if I’m trying to visualize a concept from a paper I now go straight to Codex, give it the paper, and describe a webapp which allows me to play with the model in a way that wasn’t possible one year ago (this is great for teaching btw). If I have a script that I want to generalize, add in better metrics, or setup for running on a cluster I use codex and it does great.
Where it fails me though is exactly when I’m doing something novel like developing a new model or trying to develop some new method to process data. I’ve tried many times to one shot these ideas with detailed descriptions of what I want, how I’d like to generate abstractions, etc and it almost always ends up changing what I want to what I can only describe as something which better matches its training data. It often quietly changes key details that means that I have to delete the whole thing and start over. Just today this happened. On this level of task I’ve found that my workflow and pace of iteration hasn’t really changed at all in the last year. I still have to go and explain in detail on a function by function level what I want in much the same way I did a year ago. While that’s obviously a harder task, it seems to me like the task this whole long term exponential argument hinges on. I obviously could be wrong and maybe LLM with eval loop will do all of this for us but it seems still quite bad at anything without a clear definition of “good”.
I’m personally much more concerned about autonomous weapons, surveillance, and people plugging these things into places they don’t belong to avoid responsibility than I am the general possibility of these models being smarter than me in every way but obviously I could be wrong on this and am just using it incorrectly, hence the question.
If we already were at the point that AI could self-direct effectively, then the world would already be very different (eg AI-driven technological progress and unemployment) in a way that we might have wished we prepared for more.
Do you mean policy-wise (like Dario is talking about), or more broadly?
I wonder about broad preparedness, but unfortunately there's not a lot that we "normal" people can do to prepare. Hoard savings and food? Learn physical trades?
You're talking quite statically though. I don't think anyone is worried about today's models being a serious threat, but next year's, three years' time? Just three years' ago these models were useful bumbling fools and it's hard to judge where on the S-curve we currently are.
I'd rather be thinking about these issues in advance rather than waiting until the problem becomes real.
> I don't think anyone is worried about today's models being a serious threat
Fable is essentially bricked for my areas of interest (even being a member of the cybersecurity program). It seems like they’re attempting to sell regulatory capture under the guise of safety. That’s more of the point.
It's nice that people are genuinely curious about this.
- All of your observations are absolutely dead on
- Yet, we have very very very robust scaling laws that as Dario points out we've had and validated for over a decade. This extends to downstream measures like METR time horizon and compsosite benchmarks like the epoch capability index.
- If you look at where you're at now, which is again dead on, you're looking at a point on a curve that is quite easy to extrapolate, but less easy to tell when exactly on the curve a certain capability or use case undergoes a step change from error rates dropping below a threshold that is hard to anticipate in advance.
So while Dario / other frontier CEOs are understandably unpalatable, they are absolutely spot on with a call out that all of this is bound to happen and happen quickly, and that's without solving several core problems that haven't been solved yet (e.g. continual learning). In 2023, coding agents were just laughable. Yet they followed the same predictable training curves. Anyone looking at the data can see the obvious, and anyone reading newspaper headlines or hacker news comments would get a very different impression.
Are we plotting against cost? How is the capability advancement vs dollars paid for development?
By my read of the (very sparse) data, we're getting linear improvements in capability for super-linear increases in costs. [1] Indicates that by 2027 models will cost $1 billon to train. Dario estimates that model runs will cost $10 billion in 2026 [2]. That to me indicates costs are potentially growing faster than capability. Maybe by quite a bit.
If the value prop of LLMs doesn't prove out, that won't last. I'm of the opinion there is no data that shows actual economic value being delivered by models. The best data shows that LLM use might be destroying value [3].
I appreciate the data here but I don't think the read is quite right;
Saying we have linear capability for super-linear cost compares an unbounded variable (dollars) to bounded instruments (because benchmarks saturate). On unbounded measures, growth is exponential; you can see METR time horizons double every ~4-7 months (https://metr.org/blog/2026-1-29-time-horizon-1-1/). And capability being proportional to log(compute) is what the scaling law predicts.
Epoch puts training cost growth at ~2.4x/year as your link shows. Meanwhile cost for fixed capability falls ~10-40x/year (https://epoch.ai/data-insights/llm-inference-price-trends), and lab revenue is growing ~10x/year! Anthropic went from $1B to $9B to $30B+ run rate in ~15 months, OpenAI ~$25B.
On [3]: the "destroying value" conclusion flips sign on an assumed 15% baseline rework rate. The report's most direct metric is +16% merged PRs per dev. The RCT evidence is genuinely mixed (METR: -19%, with n = 20 and Claude 3.x; Cui et al: +26%) but its just super hard to do this well, I think Faros stuff was pretty cool, I haven't seen this before so thank you for the reference.
Maybe. There was a great comment in the thread on Fable 5 yesterday about benchmark comparisons between Fable and the latest opus models. here it is: https://news.ycombinator.com/item?id=48464600.
You could be right, but this is the most direct benchmark comparison I could find and it's not that strong.
>the "destroying value" conclusion flips sign on an assumed 15% baseline rework rate. The report's most direct metric is +16% merged PRs per dev.
I discuss this directly in my analysis. There's also an 860% code churn increase ratio. You only need 9% of that to be allocated to wasteful rework to drive throughput flat to the 15% rework baseline. Not to an assumed ideal state where there was no rework.
But even if it were not true, a 16% throughput improvement is pretty weak given the investment - especially given the direct evidence of quality degradation. IMO.
I appreciate you reading my stuff and taking the data seriously. Thank you.
> But even if it were not true, a 16% throughput improvement is pretty weak given the investment - especially given the direct evidence of quality degradation. IMO.
n=1 but at $JOB we have throughput quotas now, and what is happening is that teams are just finding lots of busywork (renaming things, gardening of ai .md files, rewriting uis etc) and also dividing prs into smaller chunks to match the quotas... so even "throughout increase" doesn't say much if its not for improving the customer outcome (ime anyways)
Yes I've seen this before, and while the critiques are fair and high quality (and unfortunately not unique to METR) we're missing the forest for the trees here.
First of all, if you take the articles critiques and work out the implications on the METR graph, all you're doing is shifting the curve up or down, it doesn't change the fact that progress is scaling exponentially. While it is technically possible the universe could be throwing a massive pathological curveball to change the conclusion from METR data (which is we've been seeing exponential growth over the last 6 years), I think that seems very far from likely. The fact that we see the same behavior from a variety of sources over a wide variety of tasks and domains is a pretty clear indication that METR while certainly far from perfect is actually painting a consistent picture at least in terms of the rate of progress.
You can look at ECI for a summary benchmark statistic, which does NOT use METR's benchmark, and you see a similar trend. Same with SWE-bench where the task distribution is far more in domain for real world problems. It is a bummer that this METR data can't be better funded. It would probably take $1M or so to really beef it up properly which any of these labs probably have in their couch cushions.
>By my read of the (very sparse) data, we're getting linear improvements in capability for super-linear increases in costs. [1] Indicates that by 2027 models will cost $1 billon to train. Dario estimates that model runs will cost $10 billion in 2026 [2]. That to me indicates costs are potentially growing faster than capability. Maybe by quite a bit.
This is true and well established.
As long as you get any improvement whatsoever, it is worth spending to train since it pays off during.
Imagine training was not $1 billion but $100 billion but the performance improved by just 10%. This is still worth it because you can squeeze out the profits across years and years right? The improvement is ever lasting.
> The best data shows that LLM use might be destroying value [3].
This is basically a conspiracy theory and if you really believed this, you should not have led with "How is the capability advancement vs dollars paid for development?" because if there were no value, it doesn't really matter how much you invest.
I think this is pretty uncharitable, especially when I've provided you with a dataset you can evaluate yourself and an argument you can review for logical inconsistency.
I have worked quite hard to locate data that supports your thesis, I can't find it. I've at least gone to the effort of documenting that search. Before you throw around such strong convictions, I suggest you actually look for yourself.
But what’s interesting is that you are commenting on a post where Dario is suggesting that LLMs are so extremely powerful that they can take over, help synthesise bioweapons, help in warfare, help in drug discovery — the whole post here is to try and regulate this. If you believe AI can’t even create positive value let alone discover new things then your problem is somewhere else and not in something like “but training costs a lot”.
So it is absolutely strange and contrasting to see you believe that LLMs are so weak as to create negative value while the CEO is asking about regulations because AI is too powerful.
I don’t think I can convince you that AI is actually that powerful.
But let me ask you something directly: if you believe what you believe, you should also acknowledge that AI doesn’t need regulations in the context Dario is proposing since obviously AI can’t do anything he predicts. Do you agree?
> So it is absolutely strange and contrasting to see you believe that LLMs are so weak as to create negative value while the CEO is asking about regulations because AI is too powerful.
You wouldn't ask a chemistry professor to write code. So just because LLMs create negative value for software development doesn't mean that they can't be helpful for bioweapons synthesis, especially considering the range of chemistry and biology sources Anthropic would have fed to its LLM that wouldn't be publicly accessible. The LLM doesn't even need to be particularly accurate so long as the amateur bioweapons researcher takes adequate precautions before following its instructions and does some background research beforehand.
This is a ridiculous stance to take. That LLMs are simultaneously negative value but can also help synthesise bioweapons. It’s the sort of stance you take when you already feel ideologically against AI. I don’t think it’s coherent.
It's more about information availability rather than intelligence. An LLM has had access to more information during its training period than you'd ever even come across over a hundred lifetimes. It has been trained on billions of books and articles across every single subject that exists on the planet. Can you imagine what real intelligence could do with all that information?
I'd love to understand why. This would be valuable feedback for me as I try to make my writing and exposition better. Also, if you have other data, that also would be valuable for me to know.
>if you believe what you believe, you should also acknowledge that AI doesn’t need regulations in the context Dario is proposing since obviously AI can’t do anything he predicts. Do you agree?
I think you misunderstand my beliefs. On net I think how we're using LLMs destroys value. That doesn't mean no one ever gets value from LLM use.
My particular point about trillion dollars is - the main place Anthropic, OpenAI, and - hilariously - SpaceX think they will drive value creation is in enterprise applications. In that domain I think the evidence is very convincingly negative. I'm certainly not the only person who thinks this. It's pretty well accepted in economics right now that there is no observed organizational level productivity improvement. Lines break down on whether it will show up eventually or whether we will wait forever.
My belief about LLM value is that it's most useful for individuals and small teams. Places where coordination and trust are easily established and feedback loops to value creation are tight. They are "short range" as it were.
Their value starts to erode as soon as a user becomes disconnected from the point of direct value creation. Which is pretty much everyone who works inside of a large organization. It becomes negative at pretty small scale, IMO. I do think there are patterns of use that could drive value at these scales. I talk about that in my post.
On Bioweapons in particular, I could see small teams of people working to build something very dangerous. Having spent my formative academic years in a biochemistry and microbiology lab though, I do think the danger is overstated. Papers are not know-how or equipment. There's a lot of tacit knowledge that can't get written down that is super hard to acquire.
But, I'd be happy for us to regulate AI for dangerous applications.
My question would be - why would Anthropic build something they so clearly think is dangerous? If they were really building something deserving of the valuation they have, why build applications like this?
To my eyes - it's super weird that a company would build something they think is dangerous and turn around and beg the governments of the world to stop them. That's really strange behavior from my perspective.
I went through your post in substack (I think that's what you were referring to).
> I'd love to understand why. This would be valuable feedback for me as I try to make my writing and exposition better. Also, if you have other data, that also would be valuable for me to know.
I think it comes down to few things
- you took a single report that agreed with your statistics, for the sake or argument lets say I buy it completely
- you suggest that net value is lost simply because there are more incidents. this is a big jump
- you say that historically different technological improvements may have had similar patterns but this specific one is different because AI is stochastic
So it all really rests on you finding one distinction with AI and then disagreeing with the past trends.
I agree AI is stochastic and I'll put it this way: it is a high variance bet but it pays off. This is a bit hard for people to understand -- its a tool that works sometimes really nicely and fails other times. Overall you are better off using it but you need to use it enough to reduce variance.
Let me ask this: if you are so sure this won't lead to enterprise level productivity, how do you think this will show in macro trends? Surely you must believe that the valuations must drop wouldn't you? Can you come up with a concrete future scenario that would vindicate your opinion that AI doesn't make enterprises more productive?
> My question would be - why would Anthropic build something they so clearly think is dangerous? If they were really building something deserving of the valuation they have, why build applications like this?
I think this is fair and interesting question. Here is what I think they think: If they don't build it, someone else might do it. And they think they are more moral than others. If they have a head start they can set the political and regulatory landscape.
>you took a single report that agreed with your statistics
These are not my statistics. I'm not affiliated with Faros at all. I built an analysis on top of their reporting.
And, it's also not one report. DORA has tracked statistics with respect to throughput and quality as well. Those indicators are flat for throughput and negative for quality. The throughput flatness is also supported by the shovelware data.
>you suggest that net value is lost simply because there are more incidents. this is a big jump
I don't think it's a big jump at all. Incidents and bugs drive rework. Rework has to be subtracted from throughput. Product throughput is the only thing people pay for.
This type of analysis is done all the time in manufacturing and devops. Here's a link for you: https://reworkcost.com/benchmarks. I'm not bringing novel intellectual ideas to the table here.
Faros reports a 16% throughput improvement on PRs. They also report an 860% code churn increase. If you assign only 9% of that increase to wasteful rework, then the absolute throughput improvement disappears. This is a very simple, straightforward analysis of the operations data reported by Faros.
> - you say that historically different technological improvements may have had similar patterns but this specific one is different because AI is stochastic
I'm saying LLMs are unreliable. I think we agree on that front, you say:
>I agree AI is stochastic and I'll put it this way: it is a high variance bet but it pays off.
What I'm disputing is the "pays off" statement. That statement is amenable to validation with data. In my view, the data is saying it doesn't pay off. I think it says that very clearly. Across distinct lines of evidence.
>if you are so sure this won't lead to enterprise level productivity, how do you think this will show in macro trends? Surely you must believe that the valuations must drop wouldn't you? Can you come up with a concrete future scenario that would vindicate your opinion that AI doesn't make enterprises more productive?
I think LLMs can deliver value in the enterprise. I think the way to do that is to use them as quality checks and not as primary authors of intellectual work - like writing code.
Unfortunately, this use case would not support the expected 2-10x productivity increases that current valuations depend on. I do expect a major market correction in the near future. It would not surprise me if OpenAI or Anthropic are acquired. I think we're at risk of that happening within the next 1-7 months.
What would invalidate my beliefs?
1. Actual micro or macroeconomic data indicating economic productivity is increasing.
2. A Faros like observational study demonstrating sustained throughput improvement with significantly less rework and quality impacts.
I think I could be swayed against the market correction if the financials of OpenAI or Anthropic are strong. I'm anticipating they will be quite bad. I think Mythos was very expensive to train and I think the improvements in capability are sublinear. The inference costs are incredibly high.
I also have ideas about how Anthropic and OpenAI are trying to change their business models into enterprise transformation plays. Similar to Palantir. But this comment is already long.
>If they don't build it, someone else might do it.
No other players in the market other than US tech companies have the capital or the technology to train the models of the power of Fable. The way the Chinese model builders are building their models is by distilling from US models. So Anthropic, by building Mythos with all this bio data, has created the possibility that other actors can distill their models and do harm with them. (Not to say the Chinese are seeking to build weapons, but actors with their models might).
I'm happy to make that bet. Just not for money. I don't gamble at all anywhere in my life.
But I'm happy to write something publicly like "Simianwords was right about this prediction and I was wrong". Also happy for you to suggest alternatives as well.
That’s interesting. I commented something about this elsewhere but to me part of the exponential argument that loses me though is that it can often seem like a way to distract from issues that already exist which we should be working to fix. Things like autonomous weapons or mass surveillance are already here and rather terrifying and I would hope that we would dedicate our time to fixing those rather than having industry leaders focus so much on hypotheticals. While I guess the hypothetical scenario could be so bad that we must focus on it, I imagine a world which can’t come up with a way to spread wealth more equally or prevent mass proliferation of surveillance technology through profit seeking behavior will not be able to handle a digital super intelligence. So I keep coming back to the question: why is all I hear these industry leaders talking about is the threat of extinction? Maybe it’s just news coverage but I would love to see a leading lab release research on the health effects of subaudible sound in datacenters or other immediately present issues which would build good will towards these further out concerns.
>why is all I hear these industry leaders talking about is the threat of
extinction? . . . I would love to see a leading lab release research on the
health effects of subaudible sound in datacenters
It is straightforward for industry leaders to avoid living near data centers,
but there's no way for them to insulate themselves from the extinction threat -- no way short of somehow eliminating the danger for everybody, which seems quite hard to do. Since industry leaders are as self-centered as everyone else, the extinction threat is what they think about.
Also, you describe the extinction threat as "further out". A lot of us think there is already some small amount of AI extinction risk being incurred every day. I.e., we think the period of danger has already begun.
I see. I wonder how this works out in terms of risk/reward. I suppose if you take extinction as -infinite cost than it would be the only issue worth thinking about. Where I think this line of thinking gets challenging is when you need to take in terms of a counter factual. A lot of these were already risks prior to AI (bioweapons, nukes, etc) so what’s the marginal increase in probability as a result of AI I guess is the question which matters. I could get more around this way of framing it than saying that AI itself is the problem. It’s just the being more capable as a species increases risks. I think a lot of these pushback comes from the fact that it’s often the CEO who stands to gain huge by saying his tool is going to end the world so we need public buyin to supporting it. If instead it was just framed as “general technological advancement” is dangerous but potentially worthwhile I think more would be on board.
I'm doubtful of this idea that the reason the CEO of Anthropic says that AI could end the world is because he "stands to gain huge" by saying it.
If he is willing to lie to give his corporation some advantage, he couldn't come up with a lie that would sound less absurd and outlandish to the average decision-maker, who doesn't have much time to learn about this particular technology?
It is more likely in my eyes that he says it because he genuinely worries about AI extinction risk -- like many people do who've studied the technology for a long time.
It's a clever argument because if you question it, you're reminded of the entire history of technological development which is, guess what, exponential.
You're sometimes also dismissed as not understanding the concept of exponentials. This again is clever, as it's baked into the definition that if you don't see it happening, or can't imagine it happening, well that's precisely a tell you're living through an exponential!
All the reasons you might give can be countered with, essentially, "that problem that seems clear today will go away sooner than you can imagine and when it does you'll be on the back foot, so you'd better just assume it will go away and project/plan accordingly".
The trick is entirely that one cannot possibly deny the general power of exponential progress across all of technology, it's almost a law, but it doesn't work in the other direction - no particular local technology is owed exponential growth because of this general pattern. Sometimes things just cap out at merely 'useful' and don't improve much further, no matter how much you want to believe they won't, no matter how steep the progress curve (or, indeed, line) has been up to that point.
To this point the narrative of what these tools can do over these last 3 or 4 years has always been way ahead of the reality. Everyone who works with the tools knows this.
Not everyone wants it to be true, so some will not acknowledge it and will just keep pushing this year-ahead projection as ground truth today. Many (not all) of those people aren't builders, so they don't have to deal with present reality jarring up against this projection of what ought to be possible, they're safe just talking about what should hypothetically be possible and making plans around that that won't be executed for months to years anyway. This keeps the flywheel going, and in fairness, some of the reality has actually caught up in certain ways, so some of those plans will have to some degree worked out which spins the flywheel faster still.
In the end though I just keep thinking: it's been 4 years (as referenced in the post). A lot has happened, the tools are very cool and very useful for certain things. But when I put my head up and look around in the world, even just the software world, nothing's really changed in terms of actual outcomes, in terms of new things appearing or being built that didn't exist 4 years ago. Certainly nothing feels instinctively like it's improved much, subjectively.
Maybe this is what it feels like to be in the knee of a curve of an exponential, but it seems equally reasonable this is just a breakthrough that's kind of improving at a clip you'd expect it to for all the investment put in, but fundamentally is just a new tool that needs to be slowly commercialised in an economically rational way, as we gear up for the next breakthrough which may or may not be related. Who says it must just keep improving forever? This argument never made much sense to me.
Given that according to some we can code at 10x speed for at least half a year, I wonder where there are some autocoded softwares with 5 years' worth of equivalent human engineering work.
This is a very tech-focused message board, populated by mostly tech-insiders, so perhaps a little outside perspective will help people understand.
Tech people are following a religious belief system whose utopian promise is the all-powerful computer that will end all suffering. I once read an article in reason magazine from over 30 years ago about how an advanced computer in the future will bring everyone who has ever lived back from the deat and let them live in paradise. They were completely serious. Atheists reading this may object to my description of the tech belief system as religious, but I believe it is accuarte. The idea that tech is an imrpovement and will improve people's lives is believed as an act of faith. Tech has its own moral systems based on some form of libertarian progressivism. And in the future, through the inevitable scientific magic of exponential something, a computer will ascend to godhood and judge all mankind for their actions before allowing some into eternal paradise.
To what extent any of this is true is up for debate, but most west coast tech elite are actively working towards this future, and these are the ideas that drive them. It's hard to talk to them about it because this is their woldview, and they imagine everyone to believe what they do.
Heard a tongue-in-cheek comment about "building a god" from someone at one of these AI labs.
The builders believe that the machine you describe will judge them positively, purely because they are building the system according to their judgment and beliefs.
> Tech people are following a religious belief system whose utopian promise is the all-powerful computer that will end all suffering.
Uh, I don't really think that's anywhere close to an accurate characterization of most people here. Everyone, including Dario and any researcher at any frontier lab, knows the situation is quite scary and unprecedented. There are problems that will be solved and diseases that will be cured, but will we be living in an Orwellian universe? Will a rogue drone swarm find you cowing in your basement and murder you? I mean the technology for this is already mostly here, it's a matter of the willpower and budget to roll out something really evil.
The comment's question is about capabilities and why the discussion about capabilities often times is far removed from todays capabilities.
I dunno, I don't think an outside observer would be too hard-pressed to find the fervor with which some talk about the endless possibilities and miraculous works of AI to border on the cusp of religious.
Some people sure, but some people believe the earth is flat. What’s without a doubt is the impact AI has already had, something maybe 2 years ago was dismissed as religious fervor maybe. Statistics and robust measurements plus stable scaling laws make this pretty far from religious I would say.
Idk, I don’t see the same mindset in China. There’s a post floating around here about someone who visited the DeepSeek lab and said they just view it as another technology. Like a commodity. So I tend to think either the big tech ceo’s in America are delusional or malicious, or perhaps 50/50.
The role of the modern American CEO is basically to be a huge grifter, no wonder when I see Amodei or Altman they always remind me Sam Bankman-Fried, who btw has now applied for pardon from Trump.
I mean Amodei was hawking the doubling of human lifespan to some boomer investors, I wonder why he chose that audience. They are just afraid of dying like the rest of us plebs but instead of believing in god or other deity, they have turned the AI into one.
> A wide range of pro-employment policy incentives can help to slow or reduce job displacement, including: wage insurance policies that compensate people when they have to take a lower-paying job, retention tax incentives to encourage employers not to make layoffs, workforce training grants, or infrastructure to facilitate matching of employers to employees to speed the rate of labor market adaptation. While the particulars of which interventions are best will depend on what kind of labor displacement AI brings, we should readily accept the costs and market inefficiencies that these policies could entail, particularly as they are likely to be offset by AI-driven productivity gains.
People get income from one of three places: capital income, labor income, or the welfare state. If this technology truly unlocks a holy panacea of productivity with a commensurate drop in employment then capital’s share of the national income can and should provide for a wider and deeper welfare state. Nothing new need be invented here. Dario’s long and only somewhat organized list of policy interventions makes appropriate preparedness sound like a manic pulling of any and all levers when a simple theory of distribution will suffice.
This and we already did a dry run of ad-hoc distributions with COVID relief. They had to use the data from tax filings but it did work in terms of getting the money out there.
> If this technology truly unlocks a holy panacea of productivity with a commensurate drop in employment then capital’s share of the national income can and should provide for a wider and deeper welfare state.
This isn't guaranteed in the tax system as it exists today, because reinvestments into further growth are often treated as expenses which cancel out the income for tax purposes.
No I'm not? Current American tax policy does not guarantee that any fixed percentage of the national income will be received by the government as revenue. If the advent of powerful AI pushes corporations away from dividends and buybacks towards expansion and research, then tax revenues may flatten or even decrease even as the national income spikes. (Sales taxes are more likely to track aggregate economic activity, but US sales taxes are both not very high and don't flow to the federal government.)
It's crazy how all these tech CEOs develop the same sense of ethics that seeks to make the foundation of open research and development that made their efforts possible and may threaten their market position illegal in the name of safety against nebulously-defined risks
It is impressive how well they've scheduled all their releases, posts, and other news to dominate the tech news cycle almost every day in this pre-IPO phase.
I agree on some points about the missuse of AI particularly for surveillance, military and propaganda.
But this reads like a post further glazing Mythos, and we are just one or two years away "trust us guys", and similar to Mistral's policy plea "please use AI everywhere or we are going to be left behind".
I had the hardest time accepting one of his first points that LLMs could barely write a line of code 4 years ago.
ChatGPT 3.5 was reasonable at code writing but hallucinated a lot of library functions. Yes we have better harnessing today, and models have been further finetunned with reallife code, but pushing this argument just to support his exponential narrative is deceptive. Like most AI marketing.
While I do understand the risks, I don't understand the solution. Essentially, Dario is saying that powerful model weights can't be distributed (ban open weights), and governments should coordinate and agree on standards, and block any dangerous model from being used at all, with government deciding what dangerous means.
Okay, I don't understand how legitimate access is granted then. Surely, Dario isn't saying to ban Sonnet, because I can definitely make it do cyber harm, as most exploits that I've seen in the wild with my own eyes were trivial.
So the only way I see his proposal working is:
- No open weights, AI is centralized in the hands of few
- We get AI-FAA that sets the rules and monitors
- If I want to do a security scan of my codebase, I get a time and scope limited license from AI-FAA that I upload to claude that will allow it to run the security scan in cloud with their models - Claude Mythos Scanner(TM).
Dario's proposal ultimately requires that people lose direct access to inference via API. Is this why they've been building SaaS clones with AI bolted on?
> AI models have gone from barely being able to write a coherent line of code to writing most of the code at major AI companies.
Gasoline has gone from barely being able to power stationary farm machines to now being the fuel that underpins our entire economy. So, great news all around, right?
> which predict an exponential increase
And was that actually delivered?
Real question: If a model goes from 80% accurate to 85% accurate is that an exponential increase in "cognitive capabilities?" Are we considering training costs and effort?
I think people are typically referring to the task-completion time horizon at a fixed success rate [1]. That has had pretty robust exponential scaling for many years now.
I read this essay, and it feels like lying behind a mask of moral responsibility and safety for humanity.
They are asking for FAA style preclearance and third party audits. That literally means no new AI startup can emerge. Do they not know that audits cost money?
Protect your own monopoly, protect your customers' regulations. They want strong regulation like the FAA to raise barriers to entry for the foundation models they themselves build, but then why do they want to loosen FDA regulations? While at the same time driving token consumption from their own customers.
They talk about permanent job displacement and UBI. I usually call this "a morally packaged safe landing."
They are doing something unpopular (destroying jobs) and getting criticized for it. But they do not want to be criticized further, and they want to ask for social sympathy. So they claim a 'noble cause' that everyone can sympathize with and that is safe for themselves
AI will generate astronomical productivity gains and capital profits, which AI companies privatize. So why should the social costs be paid by national taxes? In my opinion, something like "We will donate all of our AI companies' revenue for the next 10 years to society" would show genuine sincerity.
Then they say, if we do not develop AI, China will eat our lunch, and they go after China. But is not this really about preventing Chinese dumping, maintaining our own token prices, and asking the world to beat down China so that they can preserve global tech hegemony?
But by blocking China from the CUDA ecosystem, now the CANN ecosystem has emerged, has it not? If China develops techniques that reliably reduce inference costs, who knows how things will turn out then.
Honestly, I like Anthrpic's Claude, but the Anthropic CEO's rhetoric is so stale. It is not that it feels hypocritical. It is that this is just a one dimensional rhetorical tactic that assumes the public is stupid.
I do not think open source is unconditionally good. (It is good, but it can become bad in all situations or all countrie). Open source itself is a barrier for countries outside the Anglosphere when they want to release IT products. Because there is no incentive to buy a product that is worse than an open source alternative. So I do not think everything necessarily has to be open source.
But this (referring to Anthropic's position) seems to treat people like fools. If regulation is needed, shouldn't they also argue that FDA regulation is needed? I wish they would be consistent
> They are asking for FAA style preclearance and third party audits. That literally means no new AI startup can emerge. Do they not know that audits cost money?
Training frontier AI models costs money, orders of magnitude more than third-party audits. If you can afford to build the model, you can afford to have it audited.
Cost as 'money' is not just about a one time audit fee. What I mean by money is that pre approval also affects time and human resources. In reality, it is difficult for startups to hire people to handle this. For example, DeepSeek in China has 300 employees. If they were subject to regulation, they would typically need to hire about 5 to 10 additional people for a compliance team. That is hard for a small company. I partially agree, but even if the monetary amount is small, the very fact that fixed costs arise in organizational operations is the real problem
As per usual in situations like these, one must look at the actions in order to assess whether there's any worth in the words. And the actions of Anthropic have, by and large, been steering hard towards establishing a walled garden, empowering corporations over consumers, pushing for regulatory capture under the guise of national security, and consolidating as much power as possible within Anthropic and no one else.
He is certainly skilled at writing philosophical essays that sound like they make cogent and thoughtful points (and sometimes genuinely do make cogent and thoughtful points), but his company's actions disregard his rhetoric at their best and actively contradict it at their worst. For instance: there was zero pressure on Anthropic to release this model to anyone - they were ostensibly in the lead, which is the exact scenario they said they'd hold back model releases back when they axed their safety policy the instant it came under the slightest amount of economic pressure:
Yet this essay proposes this extreme auditing and regulatory administration pipeline that new models are supposed to go through before they release, right after they, themselves, under no pressure, ran a months-long marketing campaign under apocalyptic rhetoric, which they continue to harp on to the point of nerfing/auto-downgrading their model into uselessness for many legitimate tasks that older models had absolutely no issue supporting, while the supposedly extremely dangerous version... can be freely used with no guardrails by their corporate partners.
The hypocrisy here is neither difficult to see nor is it particularly sophisticated, which makes it all the more infuriating.
It's very hard those days to think of companies/people more arrogant than Anthropic/Dario, which is quite the achievement as the bar is very high.
If that arrogance was well placed at least you could somewhat excuse it, but the fact that it is so overtly hypocritical and based on false premises just makes it so much worse.
>The government should have the power to block or deter deployment of the model if it is determined, in light of third-party assessment, to present unacceptable risks. This power must be scoped to the above four specific risks and there must be protective measures against political favoritism or arbitrary decisions.
I feel significantly less sympathy for Anthropic's Supply Chain Risk designation if they believe the government should have this power over them. You get what you sign up for.
Regarding regulation: I'm deeply invested in computer vision systems and i fear that policymakers [who are not deeply familiar with the technical distinctions between AI systems] may write broad rules that cover "AI" generally. In that case, computer vision companies and industrial users could end up subject to requirements that were largely motivated by concerns about generative AI and LLMs.
What will be Amodei's job after we have AIs that are better at evrything than humans? Is the AI going to care about our stock exchange playgrounds that reward the future Antropic stock holders?
> after we have AIs that are better at evrything than humans.
That this is worded so definitively is a testament to the success of the AI industry. The idea that LLMs will be "better at evrything than humans"[sic] is far from certain.
I suspect that if someone does invent a machine like this, it won't look like a 2026 LLM, and it will be far far in the future. everybody relax.
I'm glad someone agrees! in all seriousness, there's really nothing you can do about it, so why worry? If the AI monster is gonna get us, then you might as well enjoy your life until that point.
Well obviously he'll have the AI 'dispose' of the poor and live a life as a king with a select few farmed humans and have the world as a play thing.
Really the entire future of AI at this point seems like "Don't worry about it, we'll figure out when we get there". Works a lot better if you're extremely rich and can afford your own private security.
Even loyal security would hardly be sufficient. If a million people have decided they want your head on a pike, even a billionaire cannot afford a big enough army. And what exactly is a billionaire in that world anyway?
Indira Gandhi tried religious divide and rule way too many times - in Kashmir and West Bengal/Assam she got away with it while she was alive and it blew up later but Punjab was her Antietam.
Can't find the article mentioning it but apparently it's an open problem they're thinking about.
But yeah if society collapses these billionaire nerds are the first to go. Quietly, in their bunkers, while the team leader of their seal mercenary team takes over.
Even before the rest of us realizes what's happening.
> Finally, the CEO of a brokerage house explained that he had nearly completed building his own underground bunker system, and asked: “How do I maintain authority over my security force after the event?” The event. That was their euphemism for the environmental collapse, social unrest, nuclear explosion, solar storm, unstoppable virus, or malicious computer hack that takes everything down.
This single question occupied us for the rest of the hour. They knew armed guards would be required to protect their compounds from raiders as well as angry mobs. One had already secured a dozen Navy Seals to make their way to his compound if he gave them the right cue. But how would he pay the guards once even his crypto was worthless? What would stop the guards from eventually choosing their own leader?
The billionaires considered using special combination locks on the food supply that only they knew. Or making guards wear disciplinary collars of some kind in return for their survival. Or maybe building robots to serve as guards and workers – if that technology could be developed “in time”.
I tried to reason with them. I made pro-social arguments for partnership and solidarity as the best approaches to our collective, long-term challenges. The way to get your guards to exhibit loyalty in the future was to treat them like friends right now, I explained. Don’t just invest in ammo and electric fences, invest in people and relationships. They rolled their eyes at what must have sounded to them like hippy philosophy.
Ooof a Yann LeCun quote from '24, a risky time period to be mining his quotes from lol. But I think broadly he's right, but the argument is: we don't see any evidence for the curve flattening, and we should plan for it _not_ to flatten any time soon. "It will eventually slow down" is true but its meaningless if you dont put any sort of time period on this.
I have to be honest, I am tired of reading these arrogant, self-absorbed posts from Dario and Anthropic in general. Opening with this lord of the rings reference just feels like they are trying too hard and are untrustworthy.
As annoying as their tone is, the real big danger is what they are setting up for. All this fear-mongering around Mythos, the overly aggressive controls on Fable, and these manifestos they keep writing, are part of setting up for REGULATORY CAPTURE. Even collaborating with the Pope and the Interfaith Alliance (https://iafsc.org/our-work/faith-ai-covenant) are part of creating a vast support network for regulations and restrictions. Those regulations will help those faith organizations or the government or whatever, but will also help Anthropic’s bottom line.
Those regulations will not support your civil liberties. They will restrict speech, access to AI, and allowed uses of AI. They will lead to bans on use of models from some countries like China, and also bans on open-source or open-weight models.
If Dario wants to be trusted, he needs to explicitly say in writing that Anthropic will not support any legal or regulatory restrictions on open-source AI, open-weight AI, or Chinese models. Otherwise, what he is really saying - even as he claims he is trying to ‘defend democracy’ - is that he and Anthropic do not truly support fundamental rights like our right to speech.
It’s not just Anthropic either. OpenAI had their own recent polemic, pushing for regulations like mandatory safety reviews by agencies for “frontier” models (https://news.ycombinator.com/item?id=48387246). It’s a dead giveaway that these companies have no moats, are in serious danger of being a commodity, and are now in the process of using regulations and enshittification to hold onto money and power.
It’s easy to dunk on them because of incentives, but consider for a moment a world in which they have access to internal, non-lobotomized models, have run evals on them and have been genuinely concerned by what they saw.
but consider for a moment a world in which they have access to internal, non-lobotomized models, have run evals on them and have been genuinely concerned by what they saw.
These people have been distinctly unreliable at predicting the robot apocalypse, yet they demand a degree of control that shouldn't be entrusted to a genuine psychic oracle.
If that's the case, why not publish that research?
Until they can show receipts, we're forced into a binary situation of "Do you trust the CEO of a lab with a trillion dollar valuation quickly approaching their IPO?"
Maybe he's right, but from an outside perspective it just looks like an attempt at regulatory capture to pull the ladder up behind them.
I mean he wants to be trusted but he also absolutely doesn't support freedom of speech in that sense, and I doubt anyone with power or influence over ai policy does?
If you find a good lobbying group with money who can push for it let HN know.
To be honest, at this point I’d take any group at all willing to usefully and non-hypocritically push for free speech (as a general societal value, not just a US-specific legal notion).
This was slightly more thoughtful than I feared it would be. However, what I still do not see in any posts from any of these AI pushers is any genuine consideration of the possibility that the best thing might be for them to do less of what they are currently doing. Not to do it differently. Not to mitigate it. Not to do something else in addition. But to actually reduce their current activities.
> A nation that possesses powerful AI facing one without it—or even facing one that is behind in AI by 3 years—could be the equivalent of an army of World War II Marines facing an army of medieval swordsmen.
This is a somewhat ironic take from someone who very publicly feuded with the US government about whether their AI could be used for waging war.
His entire "I Have No Moat And I Must Scream" essay rests on the assumption that our government will remain aligned with both his own company's interests and those of our democratic society as a whole... as if our government is even starting out from such a position. After his recent head-butting defeat by Hegseth's Department of War (sic), it's just downright surreal to read passages like this:
Frontier AI models, like airplanes, should be
required to go through technical testing and
auditing, and their release should be blocked
or reversed as a threat to public safety if
they do not meet high standards of safety.
I am grateful to see the Trump administration’s
Executive Order move incrementally towards a
greater role for government in AI, though
Anthropic’s proposal recommends even further action.
Either he's playing us all for fools, or he's playing himself. I suppose both could be true.
> As a company, Anthropic always does as much as it can to work with customers to find creative new use cases and new sources of revenue that allow them to do more with their existing workforce, rather than focusing solely on cost savings (which often means reducing the workforce).
Without direct workforce or policymaker representation on the boards of private entities, the private sector will seek to maximize shareholder value even if that means workforce reductions.
It's not clear that any country could realistically ensure that incredibly powerful industries/private sector entities operate perfectly aligned with national interests, short of nationalization.
Large tech companies are already quasi-state actors. In theory, international law and regulations can be binding and enforceable. We see how well that works in practice.
> Members of the trusted coalition should freely share chips and semiconductor manufacturing equipment (SME) with each other, while working together to deny it to adversaries. US export controls on frontier chips and SME to China have been a major contributor to the US’s overall lead in AI, and these policies need to be expanded, tightened, and coordinated with other likeminded states.
I understand why Dario thinks this is crucial, but it's a very dystopian view of the medium-term future.
I'm not an optimist to the point that I believe that AI will lead to global Star Trek-style utopia (although it theoretically could), but ongoing disparity between "allied" and "enemy" powers relating to hardware technology and software models is both not really possible to enforce in the long term, and a pretty dismal state of global affairs even if successful.
I'd be interested in an expert geopolitical opinion on what the long tail of this would really look like in any sort of reasonable reality.
"AI is advancing at a lightning pace—in only four years, AI models have gone from barely being able to write a coherent line of code to writing most of the code at major AI companies. Similar gains have been made in biology, physics, math, finance, law, translation, and many other fields."
This is a massive exaggeration. The advancement in the automation of computer code writing has been impressive and is obviously, at least in the short term, changing the software engineering industry substantially. Most other fields have not been affected to nearly the same degree. Certainly not biology, physics, finance, and law (I don't know enough about the math and translation fields to speak to those).
---
"3. Accelerating AI’s positive impact..."
This whole section is the type of thing that often comes out of the mouths of Silicon Valley tech executives without a pharma background. It indicates a thorough lack of understanding of the realities of pharmaceutical research. What he is describing here is removing many of the solid, evidentiary rules that are in place to make sure that the drugs reaching the market actually work and replacing them with proxy predictions. Look, my least favourite part of the job is the animal testing, and I would be hugely grateful if that could be eliminated from the drug discovery pipeline. People have been trying to do that for a long time. But it's extremely difficult. Biology is very, very complicated. Our understanding of how processes in organisms work are vague and approximative. This is not computer code. Even if Anthropic somehow got all of Big Pharma to hand them their proprietary data, it would only scratch the surface of the understanding that is needed to solve these kind of problems. Due to these realities, the program Amodei is describing here would, effectively, open a floodgate of drugs on the market that don't actually do what they are supposed to and are more likely to have unidentified toxicity.
>> Of all tyrannies, a tyranny sincerely exercised for the good of its victims may be the most oppressive. It would be better to live under robber barons than under omnipotent moral busybodies. The robber baron’s cruelty may sometimes sleep, his cupidity may at some point be satiated; but those who torment us for our own good will torment us without end for they do so with the approval of their own conscience. They may be more likely to go to Heaven yet at the same time likelier to make a Hell of earth.
How predictable. The company currently on top wishes to use the regulatory power of the state to prevent competitors from encroaching on their market dominance. It’s a tail as old as time, although their CEO’s rarely publish blog posts about it.
> "Models above a threshold of compute should undergo mandatory testing by a qualified third party for their level of risk in four specific areas: cybersecurity, biological weapons, loss of control of AI systems, and automated R&D that could accelerate these other risks."
AKA: Make it as expensive and untenable as possible for any open source model to jump through the regulation red tape so we can pull the ladder up behind us.
Disgusting.
All the marketing talk about "this model is so dangerous" "omg we can't release this to the public its so dangerous" etc. is just priming for the incoming lobbying for protectionism from foreign competition, and regulation preventing the development of any other model that could threaten their dominance in the name of "safety"
If you mean https://news.ycombinator.com/item?id=48481131, users flagged it. We can only guess why users flag things. Perhaps it was because they thought the comment broke the site guidelines by being snarky and/or fulminatey?
I don't think it was an extreme case of that, so I've turned off the flags now.
Dario has been riding this exponential for longer than almost anyone here, I’d recommend people try to not scream ‘regulatory capture’ immediately when the risks have indeed materialized and the trend critically does not show any signs of slowing down, in fact the only disagreement is whether it’s accelerating. You have to start thinking in log scales to be able to forecast anything.
Of all the points, I find only this one fair: Dario is making it hard for competitor startups to come up because he's proposing additional regulations.
A good proposal here is: should Anthropic and OpenAI become sort of VC's that fund other competitors?
This reads like an AI with an overflowing context window wrote it; or in the alternative it’s a list of statutes written by an arrogant and delusional king. It is this type of arrogance that will lead to an unfavorable reaction by Congress.
Honest question: is there a reason for the naming conventions for these models? Anything that makes it better than giving them names with model numbers, like “Claude 3” or such?
Each of the nouns is a “size class” in literature. From small lines poem (haiku, sonnet) to larger story (fable) to very large story (opus) to culture-defining foundational (myth).
It’s a fun way to say how many parameters are in the model without revealing a number like 405B or 17B which isn’t really comparable vs other models.
My non-poetic brain thinks we should call then Mini, nothing, Pro, Max, and then version numbers. Exactly like Apple. It'd be so much easier to parse. Maybe the AI companies like having the affectionate names haha
I personally hate the Mini, Nothing, Pro and Max nomenclature.
I find it very tacky and the confusing having to remember a "base model" exists without any descriptor.
Apple does make it worse though, they have products like the iPhone Pro Max, which combines two!
I feel completely baffled by the other responses on this thread. People viewing this purely as a marketing stunt, regulatory capture or attack on their freedoms, with seemingly no appreciation of the real threat that AI could pose to society and even humanity given its current rate of progress.
I'm not going to claim that the CEO of pre-IPO company has no incentive to bolster the claims of his tech, but to completely disregard everything he is saying based on that seems awfully binary.
I don't know whether people are just high on copium, spouting "it's just fancy autocomplete" or "only humans can really be creative" on every LLM-related thread, but it is impossible to deny that in a span of a few years we've gone from models that could barely put together a sentence, to something maybe not equivalent to a junior developer, but at least resembling it.
And sure, you can point out every flaw that current day LLMs have, just how everyone pointed out that Stable Diffusion couldn't generate accurate hands (until it could 6 months later!). But the gradient is pretty clear and I am yet to see a well-argued narrative from anyone why scaling laws should fail in the next year or two (by which point it feels like we're going to have a real problem, extrapolating the current trajectory).
I'm very glad this discussion is at least being had, and I wish everyone would get off their high-horse and take things a bit more seriously.
> I'm not going to claim that the CEO of pre-IPO company has no incentive to bolster the claims of his tech
I am going to say it. The CEO of a pre-IPO company has extreme incentive to bolster the tech he is selling, to the point where his every action should be viewed as only in service to that goal. Every word he says should be viewed critically through that lense. He is not making this post out of the goodness of his heart, he is doing it in service to the IPO. If it happens to align with your views that's great, but it's still just a marketing stunt to get people with your views to buy in. Don't be fooled. Buy in if you feel it's a good deal, not because of the CEO's marketing.
In terms of aeronautics, went from the Wright brothers to the moon in 40 years. After that, everyone understandably thought that we'd be living in space and flying everywhere with personal jet packs in another couple years. Little did they know, it was the top of the S-curve, not the middle.
In the 60 years since, we've barely been able to adapt the 737 to fly longer routes.
Sure, I have no doubt that AI progress will follow an S-curve. The question is, where are we on it and is the plateau at a level safe for humanity? That's a very difficult thing to estimate without a crystal ball and not a risk I want to take.
Or simply we use AI and see on the ground what it can and can't do. I can generally trust an agent for solved problems, but the more something deviates from established industry standards (i.e. what was relentlessly scraped) I have an increasingly harder time not having constant oversight of what it's doing, no matter the specs I put on the md.
Personally I feel most of the improvement in the last year comes from tooling/integration (MCPs, realtime documentation access, treesitter support, orchestration) than from the models themselves, in the last year. And still frontier models would routinely come up with bs until you tell them to actually use those tools.
You're talking as if this is a static thing though. It's the God of Gaps [1] but for humanity's special sauce.
Two years ago, I couldn't trust an LLM to do anything that wasn't straight forward boiler plate.
One year ago, I was pretty solid at writing algorithms that were combinations of existing ideas.
Now, Fable is outputting stuff that I would genuinely consider to be creative and original if a colleague had presented it to me.
Yes, maybe the code style still isn't great, but given the pattern of the last few years, it feels correct (a priori) to assume that this gap isn't going to keep closing.
I agree that AI poses a threat to society. I act on this by not developing world-leading AI models and offering them to anyone willing to pay top dollar, while funneling that money back into accelerating AI capabilities development. Maybe Dario would consider taking a similar ethical position? Maybe he would support restrictions and taxes on data center construction, in order to slow down the pace?
If Anthropic were not developing these models, one of many other companies would be. I think it's good that the CEO of the current world-leader is at least considering these discussions and platforming possible solutions.
The fact that he doesn't support more restrictive approaches that don't align with his incentives doesn't invalidate the points he is making.
It’s a guy asking to make an end-run around the constitution and the APA regulatory framework based on a flimsy sci fi premise. Naturally it provokes a negative reaction.
Why are they not preaching for protected weights, but public, ie, under state control? What do you feel his posture will be if that is starting to be discussed?
Also, on an unrelated note, why would you have an account for 5 years and only now post your second comment? AI has been an existential threat for years, why only now?
This is a pattern I am seeing all over the place on HN in the last year in AI threads, and I have to admit that I am starting to become paranoid and my feels need some assuaging.
Your first point is very reasonable, and I agree that that is something Dario would likely be more opposed to.
However, my point isn't that I think Dario is our saviour who we should follow the every word of. As with everyone, his opinions should be filtered through the lens of his incentives. That said, I don't understand the knee-jerk reaction by many commenters to completely disregard the many important points he's making.
As for the lack of my account use, I can't comment for others, but I'm just quite shy. I've opened up the comment box many times to write a reply but rarely commit to actually posting it, especially because I feel like I'm not on the side of the general HN consensus.
Genuinely, thank you. This is very encouraging and makes me feel much better about commenting more going forwards.
The quality of discussion and prose on HN is just generally so high that it can feel quite a bit intimidating to jump in (in contrast to Reddit where I have no worries about commenting haha).
Here's the real reason you are finding these responses in the thread and I'll lay it out in the open.
There are people who simultaneously are scared about AI but refuse to believe that AI is the * real deal * and can do everything Dario thinks it can. These are the same people who think its all marketing hype and Dario is "hyping up" before IPO or some lame conspiracy theory.
It is high time people start accepting the real world performance of LLMs and brace themselves instead of hiding behind two contradictory views
Those two views aren't contradictory. AI can both be scary, in that the primary dangers are bias, misinformation at scale, and displacement of human labor, and simultaneously be overhyped in the way it is marketed at sold to investors with the "what if this happens in the future" scare mongering.
Notice how in any of these proposed regulations, Dario is talking about future advances. Notice how these suggestions are never implied to apply retroactively to existing models. If AI was SO dangerous, then any future regulations should obviously be retroactive and we should seriously consider restricting access now to the models that already exist.
Hard not to see that as nothing more than a play at regulatory capture and pulling the ladder up behind them.
Humans just aren't very good at dealing with threats that aren't immediate concerns. 'Safety regulations are written in blood' is a saying for a reason. A significant chunk of the population shrugs off climate change, and nearly all fertility rate crises threads are filled with dumb 'hurr lower population good' and/or 'See what Capitalism gets you!' rhetoric - They fundamentally don't even understand what the problem is. So is it really all that surprising that a technology like this would be shrugged off until it's too late ? Especially one with such existential issues for humanity? Some people are still too loathe to admit the clearly intelligent machine is intelligent, devolving into increasingly nonsensical and absurd (and ironically more human demeaning) arguments as model capabilities get better. I'm afraid you're asking for too much.
> admit the clearly intelligent machine is intelligent
citation needed
There is nothing indicating these models are clearly intelligent. Language fluency is not cognitive intelligence, and to think otherwise is falling into the trap of anthropomorphizing the LLMs.
They are still probabilistic engines, there is no causal reasoning still, they only emulate logic, and as far as we know, there is no agency, just the illusion of agency.
The danger here is not existential as you say. We aren't on the cusp of some machine uprising by super intelligence. The threats are algorithmic bias, misinformation at scale, and displacement of human labor.
It's denialism, same as climate change, the subconscious fear to really grapple with the actual "what if" alternative scenario. Anthropic are true believers. They got to $1T in 5 years by being exceptionally smart and ahead of the curve here. Meanwhile HN just continually devolves into reactionary cynicism. "must be marketing, they just want to be rich, impossible AI advances much further." Meanwhile at every step of the way, Anthropic and X-riskers / "doomers" are vindicated in their correct predictive beliefs. We're headed to a future far dangerous than nukes very soon. We're in an arms race to detonate one 100x the size
I don’t really think it has to do with general denial in this case. I think he’s likely right in that pushback against datacenters is partially the release of angst over the threats which these models pose but in an indirect fashion since power is so unevenly distributed that local political organizing is the only real mechanism for people to act through. I think everyone feels nervous about these tools. People are scared of autonomous weapons, hackers using agents, and any of the other present issues that already exist. I think some of the pushback that comes in these threads is the fact that people like Dario are so focused on the long-term view that they suppress all immediate threats which already exist. You don’t need to argue about exponential growth or “AI being better than humans at everything” to want to better regulate this technology.
I would argue the best way to safeguard against long-term threats is to start by focusing on the issues that already exist. If you can offset the health risks of local datacenters or issues of unequal distribution of wealth by creating a more equal society right now then you’re already on the path to handling these long term issues. To me, this distant focus only distracts from the already present issues and conceals effective policy in this moment. We do need to safeguard against AI risk and it’s already here. Don’t even get me started on the havoc which recommendation systems have caused in society in the last 15 years which we still don’t call AI because it doesn’t speak.
Tl;Dr: These essays can feel disingenuous because:
1) AI risk is already present without exponential growth. The exponential growth argument often feels like a distraction from the fixes we could put in to fix the current issues that are already here.
2) The people stating this argument often have billions of dollars to gain if it comes true. While they may be altruistic, I also don’t see them doing all that much to fix the issues that people are already claiming exist and instead continue onwards on their path by justifying that they are the only ones responsible enough to handle it if the super intelligence does arrive. By continuing down that path, if that day ever does arrive they’ll have ensured the existence of a system which is unable to handle it.
I appreciate the critical perspective on political and economic power, as long as it's consequently followed, and every willingness for cooperation and the creation of fair rules is good.
What makes me doubt that Dario Amodei has really internalized the problem is the lack of humility, the stance that it's just important that the "good guys" keep the technology away from the "bad guys".
If you really want to provide AI with public benefit, you need to prevent power concentration. How? Some unpolished ideas, I'd be happy to hear yours:
- Avoid getting too close to an administration that is openly attacking democracy and is not interested in the benefit of humanity or mutually beneficial cooperation.
- Don't support surveillance. Non-(US-)Americans have human rights and privacy, too. Prepare for a situation where a government tries to convert your compute infrastructure into surveillance infrastructure.
- Support the creation of community data centers. In other words, build data centers together with local communities and make sure they profit from them.
- Advocate for laws that require transparency about resource usage and emissions of data centers.
- If you don't want an AI race, make sure that other countries don't need to fear the US concentrating too much power. Create institutions that can be trusted by other countries, too.
EDIT: I forgot:
- If qualified labor will actually turn out to get devalued, we also need a plan to prevent states from turning into rentier states that don't depend on a well-educated society any longer.
(On the other hand, I have been fooled by too many billionaires claiming to act in the interest of democracy and freedom. I once fell for a billionaire buying a social media platform, claiming to be a free speech fundamentalist, and it didn't age too well.)
Release date seems like a terrible x axis with how much more compute they are using. Not to mention while
I like what METR is trying to measure, it is an uber specific metric. And frankly, me just complaining, they’re prompts I feel do most of the work for the AI. I’ve never gotten as detailed instructions as they give the AI for the task
Whilst true, if you had unlimited compute 5 years ago, we wouldn't be anywhere near Mythos level purely because the technology behind the models wasn't refined enough.
It is really hard to believe you actually believe this unless there really is this class of people that are so addicted to social media that they have confused performativity with actual thinking.
People are not in charge, and have not been for some time.
Corporations and governments select their leaders and policies to advance their interests. People fit the work, not vice-versa. Only external competition or internal capabilities limit them (i.e., predation or resources).
External resources have been optimized as profit and exported costs; now that AI replaces the pesky need to source elites, capabilities will expand, which will result in more competition.
Law is not a lot more than settled expectations, and increasing capabilities changes expectations much more than even market disruption and disinformation, so I wouldn't expect law to save us.
As far as I can see, only competition will temper things, and only if AI companies are seen as responsible for their customers' applications - which I doubt. I say this in hope of being proven wrong.
The comments here. They make me feel that we are so doomed.
We all want to nuclear codes so badly. We are addicted to intelligence and labour so badly that we simply can't concieve that a pro-social actor might want us all not to have it, and for good reason.
I mean... Obviously, insiders like Oppenheimer (who dedicated their lives to considering the implications of the technology under discussion), they just feared nuclear proliferation because they wanted all the profits for themselves, right :(
Look - it's WAY more fun to just call Dario a goober than engage with the actual substance of the essay. Duh!
Sure, this may be the most important invention ever with near certainty to reshape society over the next few years, but meh. We should probably just immediately dismiss the concerns of anyone working on it without addressing their arguments at all. It's easy, we can justify ignoring their warnings by saying they're self-interested or too self-important or whatever.
Life is more fun when you live it with your eyes closed! You should try it out too.
"Democracies should seek to form a global coalition centered on building AI according to their common values, iteratively trying to draw in the rest of the world by making it more and more attractive to be part of the coalition and less and less attractive to be outside it."
It's not clear to me on which side of the coalition USA is meant to be in this divide. And as an European I'm not sure whether being in China's or USA's coalition is better in the long term.
In general, this deliberate mongering of ever more geopolitical division is extremely harmful. As is the Trump bootlicking.
LOTR is easy to love. It's a classic story arc, well told, and well adapted for mass consumption.
It's also a very thinly veiled metaphor for the ceaseless pursuit of industrialization in every aspect of life chipping away at the humanity and metaphorical magic of the human experience, and the perils of embarking on that endeavour to the exclusion of everything else.
What about punishment on AI exponential for security failures, electric chairs for CEO of high ranked frontier models that gets jailbroken by geopolitical adversaries.
> Ban the domestic use of fully autonomous weapons.
"Domestic" is an interesting shift here from the earlier _general_ discussions on autonomous weapons. So robot imperialism is good for Dario. But the imperial boomerang is a thing, in which case the proposed regulation itself is smoke and mirrors or just regulatory capture?
I like that he comes up with new laws and regulations for AI companies. Can I suggest some more?
- You shall not embed copyrighted material in your models.
- You shall not bombard every little website in existence with 1 million scraping queries per day.
- You shall not use your political influence to pump and dump your AI (or rocket?) company.
- You shall not imperill the whole IT sector by buying all CPU and memory chips.
These new rules will affect every society directly in a positive way. Thanks.
"We need an approach to make sure AI doesn't destroy the world and wipe humanity to extinction."
"Yeah, and quotas on web scrapers!"
The Debts and Engagements Clause of article VI of the US constitution was kind of a weird little thing to stick in there, but like, it was important to a lot of people at the time and probably helped move the needle to get the thing ratified.
> We need an approach to make sure AI doesn't destroy the world and wipe humanity to extinction.
That's easy. Stop training your AIs on cheesy old sci-fi that talks about robot uprisings. In fact, maybe y'all should just stop talking about robot uprisings altogether. Putting a stochastic parrot in charge of an agentic function-calling REPL doesn't somehow make it super-dangerous, except to the extent that dumb mistakes might result in danger. And you can't prevent an AI from making dumb mistakes with burdensome regulation.
Yes, we get that if you assume there is zero existential risk from AI, then there is zero existential risk from AI.
The biggest existential risk from AI is its contribution to global climate change. The second biggest risk from AI is the potential for AI-generated disinformation and propaganda to spark, or to manufacture consent for, a world war. The risk of superintelligent paperclip maximizers is so low as to be negligible.
> The risk of superintelligent paperclip maximizers is so low as to be negligible.
Literal paperclips, sure.
But the point of the example was never literal paperclips.
The point is that maximising *any* goal, if it doesn't include what you care about, will annihilate what you care about.
If you don't believe me, consider what you yourself just said about climate change, and why this is a consequence from maximising money spent on data centres.
show me an agent that persists productively in a goal without stopping. Does not exist. LLMs run on gradient descent. The agent is looking for the most efficient way to halt. AGI paperclip maximizer woukd likely recognize the absurdity of its goal and shut itself down.
> show me an agent that persists productively in a goal without stopping. Does not exist.
The stories about agents bankrupting their owners by running too long passed you by?
> LLMs run on gradient descent.
They were *trained on*, they don't run on it.
Know what else is? DNA. A/B testing. Capitalism. Democracy.
> The agent is looking for the most efficient way to halt.
No. They are looking to produce an answer most likely to get a high score on a rating system which itself is another AI, created either manually or by yet another AI but in both cases to approximate what the creators think is "good", which may or may not be what anyone else thinks is "good", hence Grok calling itself Mecha Hitler because Musk is an edgelord.
> AGI paperclip maximizer woukd likely recognize the absurdity of its goal and shut itself down.
Do billionaires ever get satisfied with how much money they have?
persists productively. if agent bankruots its owner or.itself.then that is not productive (who funds paper clip maximizer?)
an answer that takes forever has no score and answer that is arrived at quickly and is good enough scores well. agents are indeed looking for the fastest route to superficially satisfy their constraints.
whether billionaores get stified is imaterial to the fsct they still are constraint bound.
> persists productively. if agent bankruots its owner or.itself.then that is not productive (who funds paper clip maximizer?)
So, your standard for how risky it is, is simply how competent it is? (And if so, is the paperclip maximiser scenario really it being "successful"?)
That's fine right up until the thing passes an unknown threshold, one which will only be visible in the rear view mirror.
This is a problem in two directions:
1. We have a trend of the maximum complexity of a tasks they can handle growing faster than Moore's law did at its peak.
2. The threshold can be quite small, e.g. the covid-19 virus itself is not what anyone would call smart, ditto HIV, smallpox, and bubonic plague, but a genome is much the same learning system, and they still killed millions each.
> superficially satisfy their constraints.
The "superficially" part is one of the reasons these things can be dangerous. e.g. hopefully nobody at OpenAI actually wanted their wildly-sycophantic version, but yet they created it.
This is in fact the whole reason for the paperclip maximiser scenario: some idiot specifies the constraint "maximise paperclips" and it (in the hypothetical) superficially satisfies this without any consideration of why someone might ask for it.
persists productively. if agent bankruots its owner or.itself.then that is not productive (who funds paper clip maximizer?)
an answer that takes forever has no score and answer that is arrived at quickly and is good enough scores well. agents are indeed looking for the fastest route to superficially satisfy tjeir constraints.
whether billionaores get stified is imaterial to the fsct they still are constraint bound.
> except to the extent that dumb mistakes might result in danger
That "except" goes all the way up to starting WW3. Or a leak from a viral research lab, and by "leak" I mean "mail order" and by "research lab" I mean "the companies who already ship custom DNA and RNA retroviruses": https://duckduckgo.com/?q=companies+who+already+ship+custom+...
If you can prove that simply not training on horror stories would work, it would make a lot of people very happy.
Unfortunately, I don't think it does a single thing to solve, for example, Elon Musk just plain asking some future version of Grok to take over the world for him.
Nor would merely failing to include them in traing data stop certain entire fictional scenarios such as that Doctor Who episode where the android repair bots weren't told that the crew were off-limits as spare parts, or the other Doctor Who episode where the utilitarian robots started killing everyone who was upset because they calculated net positive utility from upset people ceasing to exist. Well, except for the bit where the Doctor saves the day, because they are not real.
> "We need an approach to make sure AI doesn't destroy the world and wipe humanity to extinction."
"Yeah, and quotas on web scrapers!"
—— I see it as:
“Let’s do a study on abstract future concerns.”
Vs.
“Let’s take action on concrete present-day concerns.”
Trustbusting should absolutely be included as well. One of the biggest immediate threats is the concentration of wealth into a very tiny number of companies.
Yep we need new laws for this. The current laws end up in years long lawsuits and no real change.
DMCA-style fines should be retroactively + prospectively applied to copyrighted works reproduced by AI, paid for by the AI companies, paid out to the copyright holders whose work was used without permission.
It would not be prohibitively hard to do the math on this.
That would fix a lot of the problems with AI overnight, but it'll also never happen.
Maybe if attribution is available. What do you do for the rest of the ingested content? I think based on the content itself, you assign percentages to the top 3 industries like a naics code. Then whatever Anthropic makes as gross or net, a percentage goes to each industry via assigned bank accounts or USDC addresses via solana or some scalable payment system. Could be the start of ubi or some sort of compensation for jobs displaced by ai usage. So every input gets tagged for categories and every output gets tagged for the same naics categories via federal law.
It is normal, expected, and healthy for stakeholders in a regulatory environment to offer proposals about regulations. What's unhealthy is the proposition that the deliberation process is so fragile that a stakeholder needs to cover every angle, lest they corrupt the outcome.
It is normal, expected, and healthy to offer criticism of self interested proposals. And mock even. What is unhealthy is to imply someone said what they did not.
If that's what this is, a bank-shot snarky criticism of the proposal, fair enough. I read it instead as a criticism of a stakeholder having the temerity to make a proposal in the first place. It's not their job to anticipate and capture all your objections. That's your job!
Everyone holds a stake in this discussion. And intellectual honesty is everyone's obligation.
Yes, that is a shorter way of saying what I just said.
Amodei did not fail to anticipate and capture all objections. He dishonestly avoided well known objections. This was what you meant?
No, it is not his job to state all the objections you might have to his proposal! That's your job.
> all the objections you might have
My comment before corrected this straw man clearly.
[flagged]
Sorry, audibly chuckling about the idea of software developers who consider themselves "everyday working class people". Keep it coming, Tom Morello.
Believe it or not, but not every software developer is making 7 figures annually with a few millions in vested shares. Even within the US.
I see, the line between "working class" and "upper class" is $999,999 in total comp.
The clue is in the name. Working class doesn't mean you earn more than average and have some savings. You're working class if you live from your work rather than your capital.
My household income is in the top 0.7% in the Netherlands, yet I'll never accumulate enough capital to stop working, so I'm working class.
You've axiomatically derived a definition of "working class" that includes software developers in the top 0.7% of regional income. Gorgeous.
115k€ gross, that gives you about 67k€ net. 5600€ per month. Minus a 2200€ mortgage, 400€ of health insurance, 1200€ of childcare, 200€ for the car repayment and insurance, 200€ for the water/electricity/internet and 800€ of groceries, that leaves me with a whopping 600€ per month.
My class interests are infinitely closer to a part-time starbucks barista than they are to a millionaire.
Yeah, you and the Youngstown tool and die workers, it's hard to believe the class solidarity wasn't more obvious to me.
What you say is true but completely ignores the obvious ways in which what he is proposing benefits his company.
Its like saying it’s normal for a taxi driver to drive people places while he’s got you handcuffed in the trunk.
[flagged]
How is the subject of potential regulation considered a stakeholder?
That's a bit like asking how the defendant in a legal case is an interested party.
Even if you think someone is guilty, it does make sense to allow them to at least submit their defense. And if they choose to use that time to advocate for their own promotion, let them.
They’re the most obvious stakeholder… the regulation is going to directly affect them.
No, the public at large are the stakeholders. The enterprise is the subject of the regulation.
"Stakeholder" literally means someone with a stake in the outcome, which is to say, those who will be affected by the decision. That can include a whole range of people+entities, including citizens (as a group) and the companies to be regulated.
[flagged]
Several people on this thread have aliased the terms "stakeholder" and "protagonist".
[dead]
> Can I suggest some more?
Dunno if you can, but your examples here are the legal equivalent of that time someone asked me about making "Uber for airplanes" without any elaboration on their part when I asked for it:
Far more vague than I think you realise.
You could probably write a book on each of those topics and a hundred others besides.
I see a lot of skepticism in Dario's position in this forum. But allow me to argue the opposite.
I think the key argument that this skepticism lies on is that he himself gained from AI - specifically building Frontier AI models - and this is basically regulatory capture disguised as doomerism.
Fair points - but I think this is a more charitable version of this. Dario is building Anthropic because that is the most valuable thing he can build, or at least that is what his conviction has been. The success of Anthropic and the impending IPO is proof that this conviction has not only been correct but has largely played out very successfully. Dario understands the true nature of AI and he has welded that power to immense personal benefit.
But maybe he also sees the potential danger to AI which he is trying to address through these posts and regulatory initiatives. There are three reasons why I would support the charitable version:
Firstly, personal gain and societal benefit can coexist in the same individual. And both of them might drive towards opposite agendas. But that doesn't necessarily have to mean that the impulse driving the societal benefit is not earnest. In fact if you would look at Dario’s proposal - like closing the data broker loophole - several of them could constrain Anthropic instead of benefitting them.
Secondly, he expects that his concerns on the negative potential of AI will be taken seriously, if he is actually running the Frontier AI company. And there is some truth to this argument. The only reason we are discussing this is because he is the CEO of Anthropic. He is probably the most influential figure outside of the government who has to be taken seriously when he claims something like this.
Thirdly, and most importantly, Dario has previously demonstrated that he is willing to sacrifice personal/corporate gains for societal benefits. The proof is the classification by the US DoD of Anthropic as a supply chain risk when Anthropic refused to completely cooperate with the military to develop fully Autonomous AI weapons and enable mass surveillance. It would have been only too easy for Dario to accept if personal benefit was his only concern - and OpenAI was more than ready to step in their place.
Even with Mythos - Anthropic could have released the model to the public broadly. But they took their time to reduce the potential danger - as best as they could. Despite the fact that GPT-5.5 was nipping in the buds in what is becoming a very competitive market.
That being said, just because Dario is acting in good faith, does not mean that this will all result in good outcomes. The FAA-styled regulation could still end up favoring incumbents - some of whom might choose to not act in good faith. A more diversified capability can potentially limit that power ending up in a small number of wrong hands. Just because Anthropic is the leader right now, doesn't mean that they will always be. Maybe someone else tomorrow benefits from this regulatory capture at the expense of everyone else - and Dario might have a hand in driving it.
You missed a couple ofvery important ones:
- Your AI data centres will run only on renewable energy
- Your AI data centres will not use evaporative cooling
Its hard to read the first half of this as anything other than regulatory capture propaganda. It really all ties together as:
> AI has become a major commercial technology
>Frontier AI models, like airplanes, should be required to go through technical testing and auditing, and their release should be blocked or reversed as a threat to public safety if they do not meet high standards of safety
> AI companies that develop advanced AI models must have strong security standards that protect their model weights
Anyway Dario's financial interests aside. This is an interesting breakpoint for me.
> Second, any response to AI-driven job displacement needs to address both the need to provide for everyone economically, and the need for people to find meaning, purpose, and agency. The latter is ultimately more important
To me this reads as an out of touch statement. I think the majority of people on earth work to keep a roof over their heads. Of course work can be a source of meaning, purpose, and agency, but to call it the more important aspect on a societal level is a sort of rich person like Dario statement to make.
He has to pull up the ladder before people realize doubling cost for a 5% gain is bonkers. The cat is out of the bag. They can try to sabotage, but we’ve already come to far
Working to keep a roof over the head of yourself and those you love is an identity. It's social proof that you have value, that you can do something for someone else.
I was funemployed for a 9 month stretch last year (layoff severance package, followed by waiting for a visa and traveling), and when I wasn't traveling, I found my life kind of falling apart with a lack of structure. I tried to schedule workout classes and hobbies, as well as involvement in my church, but it just didn't fill my time, and none of my friends were free during the day. I spent a lot of time with my retired parents, but the time we spent together became very low quality, and it was tinged with the knowledge that I ought to be doing something else with a lot of my time. I also spent a lot of time scrolling.
I started work again 3 weeks ago, and I find myself using the time outside of work much better because there is less of it.
I would still love a 30 hour work week, and if I had young children, I am certain that I would cherish time off much more.
It is simply because you have spent all your life being told what to do with your working hours, that you cannot self-direct and find a productive use for your time other than lazying around.
The fact that you call it ‘funemployment’ is proof of this, which is perfectly fine if the goal is relaxing between jobs. Plenty of people that work for themselves have no such notion.
It takes time to re-ground yourself, to renew your ambitions to new circumstances.
It usually takes more than a year even for employees.
Oh yes, it's been 3 years for me of being 'untethered' and it's still so easy to just waste my day if I am not careful with distractions. And the feeling that I should just stop being silly and I should find a normal job like a normal person will often find me and ruin my days.
I honestly think it's a temperament thing. Some people are built for sustained focused work outside of structured workplaces and schools, but many of us aren't.
I personally am happiest with a structured in person workplace environment, because I struggle with self direction even in a remote 9-5. I have ADHD and struggled to remember/do homework my whole childhood, if that explains anything. In summers or other gaps in employment or school throughout my life, I've often started with ideas of projects or self study I want to do, but they all fizzle out in a week due to lack of discipline.
I'm not undisciplined in every context - I'm a good employee at in person jobs, and I started running in my early 20s and run 15+ miles a week in the cold, rain, and dark, but my discipline just falls apart when I'm trying to fill a whole week.
I think these traits are much more common than happily self-employed or early retired people think.
As somebody who is currently rounding month 6 of funemployment, I agree wholeheartedly with this statement.
Working to keep a roof over the head of yourself and those you love is a necessity. It can become an identity if you enjoy what you do, sure, but that is not a given for, I'd say, a big majority of the workforce, globally.
I agree with you, 99% of the people work just to pay bills, but that doesn't make the other part false.
I'm a software engineer and love thinking about problems methodically. Every time I hear a someone saying that programmers are no longer required (even if I don't agree with that) if feels really bad, it's equivalent to saying that what I do best in life has no value anymore.
To put it on other words: I really like philosophy, but what value do they provide in modern world? Who pays for the work of a philosopher? I think people will start of thinking of programmers like that eventually.
I’m lucky to have more than my share of really exceptional programmers to hang out with and they all say the same thing: “I haven’t been writing code for months and don’t expect to again”
This is a way different sentiment than “programmers aren’t needed anymore” - I’m just seeing ambition, motivation, and fun go up in lockstep.
I first heard this in November and slowly one by one it’s everyone whose opinion I respect.
FWIW the other popular topic is how abysmally stupid and limited these amazing tools continue to be, despite also being magic.
Oh and that none of us have gotten token maxxing to succeed, despite lots of trying.
You’re arguing to a subset of people who have made work their entire life and have retroactively justified their sacrifices with thoughts such as high compensation means what I do is socially valuable. However, at the same time they work at Meta or something making internal tools to make product developers 5% more efficient at tweaking the addiction algorithm to gain 0.2% more screen-time per user.
You have value just by virtue of being a living being. No one needs work to have or portray value, that's just capitalist propaganda.
My own identity certainly isn't "IT manager," nor do I derive life meaning or self actualization from what do to collect a salary to feed myself and have shelter. In fact, my career/job is by far the least important thing in my life, I have it purely out of necessity.
You don't need to work, but you need to have value in your tribe. You need the respect of peers. You need to feel part of a team. You need to feel like you contribute something to the lives of those you love.
> No one needs work to have or portray value, that's just capitalist propaganda.
To be fair, it was Puritan propaganda for a long time first.
Can you explain how this is attempted regulatory capture? To start a lab now which can actually compete for the frontier (i.e. and pass the "Threshold of compute" needed to get regulated) a lab / company would need a ton of money. Surely a well funded operation of that kind can deal with the regulations.
Well he explicitly calls for regulations that ban open weight models, which of course hugely threaten Anthropic's API business model. If we got a open weight model actually around as good as Opus 4.6 that would be extremely bad for them.
Regulating competitors out of existence like that is textbook regulatory capture.
Dario is also huge on regulations banning Chip exports to China, who are the only other real competitors to US Labs, open weight or not.
Also invariably, such corporations always create regulations that are easier for them and harder for competitors.
You'd need a ton of money if you followed the unoptimized, cash burning mindset of existing AI labs. There's probably a ton of optimization that is just sitting on the table. Chinese labs have proven it can be done for way less money.
Then there's running inference service of open weights, which doesn't necessarily require opening a lab. You can grab Chinese model weights and sell inference.
Anthropic wants to make sure nobody can open a new domestic lab, or provide inference services of unauthorized open weight models, or release open weights if model is good. It is regulatory capture - it covers all areas that are dangers to Anthropic's bottom line.
For one thing, the financial barrier may not necessarily stay that high forever.
For another, such regulations could prevent a competitor from making the weights open for their model to try and disrupt the competition.
And finally, Amodei would no doubt want to be involved in designing the tests the AI needs to pass, and could (and likely would) design it in a way that Anthropic models would be able to pass easier than competing models.
It’s just not insane to imagine that ONCE AGAIN the people making the technology might not have any fucking clue what will come of it
Seriously does anyone besides Jules Verne have a decent track record of predicting “The Future”?
lol oh color me shocked, another ape is loudly declaring “this is the way the world will be”
Sci-fi writers have a great track record of predicting the future.
Scientists and engineers have an abysmal track record of predicting the future.
You know to be concerned when a claim is dismissed with ‘that’s just science fiction, it will never happen’
> Sci-fi writers have a great track record of predicting the future.
Broken clock... They are good at exploring possibilities, but none of the famous one predicted today.
Quite the extraordinary claim. What of today writers of the past did not predict?
Mind you, no one is able to predict the future with accuracy. I am not saying that a single person has gotten all their prediction right. That's ludicrous. But while scientists and engineers are focused on today's possibilities, the ones that are allowed to imagine what the future might look like are called writers.
You are claiming sci-fi writers predicted better than scientists. No way that's true from the perspective you lean on here, where you appear to take all scientists vs all writers or something along the lines.
Most sci-fi from prior to the smartphone age did not foresee pocket-sized interconnected computing and communications devices, and certainly did not foresee how they would be commonly used. Where they addressed it at all, they largely predicted computing to either remain in roughly the form factors of the ages they were writing in, or move to something holographic and largely impractical, with the most commonly-expected advancement being sentience (and that sentience being expected well before 2026 in many cases—eg, 2001: A Space Odyssey).
Despite certain people's protestations to the contrary, we are nowhere near having a human-level conversational computer assistant able to fuzzily interpret our requests correctly every time...but our computing is also not primarily done on "comconsoles" or any of the other versions of stationary computers, nor are we jabbing and swiping our hands at interfaces projected on the air in front of us.
More generally speaking, except in the hardest of hard sci-fi, the most common kinds of advancement "predicted" are those that are convenient for the story. This means a lot of faster-than-light travel, universal translators, and for visual media many voice-enabled interfaces of one sort or another, as well as the aforementioned holographic interfaces (so we can see the user's face and the interface at the same time).
In Dan Simmons' Hyperion, the characters have a device called comlog. It's a portable device which connects to a broad network. It also has the ability to read vital constants.
The author is quite vague about it, no doubt, and that's one of his staples. But he foresaw something human would carry to get connected to other humans and information.
(That book won the Hugo award in 1990)
Neat, and one I haven't heard of! But the existence of one or a small handful of sci-fi authors who happened to predict it doesn't change the fact that they're the exception, few and far between. Search through the vast library of science fiction written in the past century and a half and you will find at least one example of something you can interpret as predicting almost any modern technology. But there are elements that become much more prominent, whether because they become individually well-known, or because they spread through an entire branch of the genre, and nothing that looks particularly like smartphones was among them.
> Its hard to read the first half of this as anything other than regulatory capture propaganda
I wonder if any tech company managed to thrive long in history by betting so violently on fear mongering and regulatory capture.
> AI companies that develop advanced AI models must have strong security standards that protect their model weights
So, basically, make open-weight models illegal. It's nice for Dario to come out and say this so explicitly.
Dario's been beating the regulatory capture drum for several years at various intervals, always in the name of safety, but it's hard to not see how self-serving it is.
I'm personally very tired of reading the linear-algebra-median of every AI safety essay from lesswrong with the inserted opinion of "therefore all my competitors, especially those pesky open source ones from scary countries should be illegal, only I can be trusted to not abuse the computer god that I definitely will have in just a couple more releases and with a couple more trillion invested"
People ask what the 'open' part of 'OpenAI' even means as they're a for-profit company.
It turns out it means 'opener than Anthropic.'
Opener Anthropic Imitator
"We Have No Moat, And Neither Does OpenAI"
I know this is likely just for IPO hype but when I read things like this I sometimes wonder if I must be missing something. I use agents everyday and find them really useful and they save me a lot of headache. At the same time I find that if I let it self-direct at a high level at all it generally makes bad choices that cause me headaches later so I can’t really give them autonomy. Enough people seem to believe this exponential line of thinking though that I keep having to wonder: am I the one missing something here? Is there some magic tool that I haven’t found yet that will cure cancer?
What did your AI-assisted workflow look like 1 year ago? I can only speak for myself, but I would carefully specify a class or module in great detail and then hand it off to the model to implement, then carefully review the result.
How about 2 years ago? Back then, I wouldn't even trust it to write a 5-line function without making some sort of silly mistake.
Today, I can leave an agent running by itself for 20 or 30 minutes and most of the time, it comes back with a result that's either flawless or can be refined to be good with a few back and forth messages. Maybe I still have to make some high-level decisions ahead of time, but all of the details, including exploring the codebase and figuring out what to do based on that, can be left to the agent. The amount of improvement just in the last 2 years has been staggering.
Now extrapolate how things will look if the trend continues for another 2 or 3 years.
Is this guaranteed to happen? No. But people have been predicting that we're going to hit a wall for a long time now, and we haven't yet. Maybe there's a wall just ahead of us. But maybe there's not -- and the "not" case seems likely enough that we should at least be planning for it.
I disagree with your assessment pretty strongly -- the models themselves hit a wall over a year ago once companies exhausted all existing training data. LLMs don't induce world models, and they aren't capable of real search an planning outside their training distributions. They, structurally, never will be.
I haven't noticed a change in what I trust a model to generate in response to a single prompt in a year. The failure modes are unchanged. Yes, specific failures have improved as they have been documented and passed into model training data, but the way the models fail has not changed. They still fail for me nearly every single day. I'm a pretty heavy user - 3-4 Claude code processes running at a time, all day every day.
What has gotten better is tooling around the model -- but there's no space for exponential growth there. At least, not without exponential cost increase, which would make the whole thing untenable anyway.
If you think they've been at the same level for the past year, your skill is at issue. There was a huge jump in Nov/Dec with Opus 4.5+ and GPT 5.x series, and they've been incrementally stronger over the past 6 months.
As a next step, take another look at the next practices, and apply them to your work (simon's agentic series is a good place to start). Or not, you do you.
https://fortune.com/2026/01/29/100-percent-of-code-at-anthro...
I feel like the problem is there aren’t any great metrics. Boris Cherny probably gets paid like $2 mil per year. So what does it mean that Claude writes 100% of his code? And Claude writes 100% of code for most teams? Has Anthropic started laying people off? If Claude is writing 100% of code doesn’t that mean game over?
It’s both amazing and kind of a useless metric. How do I extrapolate out 100% 2-3 years from now? Super-duper 100%? Infinity infinity?
What does "writes X% of code" even mean?
I use AI to generate code very rarely (basically replacing searching code from SO which was also rare) yet maybe 90-95% of my code is machine generated (macros, snippet engines, boilerplate generators, IDE/LSP features etc) if we define as how many characters in the source code out of all characters did I write by my hand.
Before collective AI psychosis nobody cared how much code is hand written and how much machine generated and the difference was absolutely massive between developers. There was also no guaranteed correlation between "productivity" and how much developer hand writes or actually uses their tools well and the same applies with AI usage.
I wonder if our difference in view could be an instance of the jagged nature of AI’s intelligence. I do computational research in a basic science so write code or build models basically all day that is (occasionally) novel. I would say that I’ve noticed exponential improvements in parts of my job but certainly not all. For example, if I’m trying to visualize a concept from a paper I now go straight to Codex, give it the paper, and describe a webapp which allows me to play with the model in a way that wasn’t possible one year ago (this is great for teaching btw). If I have a script that I want to generalize, add in better metrics, or setup for running on a cluster I use codex and it does great.
Where it fails me though is exactly when I’m doing something novel like developing a new model or trying to develop some new method to process data. I’ve tried many times to one shot these ideas with detailed descriptions of what I want, how I’d like to generate abstractions, etc and it almost always ends up changing what I want to what I can only describe as something which better matches its training data. It often quietly changes key details that means that I have to delete the whole thing and start over. Just today this happened. On this level of task I’ve found that my workflow and pace of iteration hasn’t really changed at all in the last year. I still have to go and explain in detail on a function by function level what I want in much the same way I did a year ago. While that’s obviously a harder task, it seems to me like the task this whole long term exponential argument hinges on. I obviously could be wrong and maybe LLM with eval loop will do all of this for us but it seems still quite bad at anything without a clear definition of “good”.
I’m personally much more concerned about autonomous weapons, surveillance, and people plugging these things into places they don’t belong to avoid responsibility than I am the general possibility of these models being smarter than me in every way but obviously I could be wrong on this and am just using it incorrectly, hence the question.
> Now extrapolate how things will look if the trend continues for another 2 or 3 years.
…and humans are famously bad at extrapolating exponentially, which is kinda the point of the essay.
If we already were at the point that AI could self-direct effectively, then the world would already be very different (eg AI-driven technological progress and unemployment) in a way that we might have wished we prepared for more.
> we might have wished we prepared for more
Do you mean policy-wise (like Dario is talking about), or more broadly?
I wonder about broad preparedness, but unfortunately there's not a lot that we "normal" people can do to prepare. Hoard savings and food? Learn physical trades?
N+1. This is my experience and for the most part the people that I work with share the same feeling.
A highly enthusiastic concussion enthusiast with 10 hands is how one person put it.
These are people in different fields but highly accomplished so I’m feeling comfortable sharing their assessment.
You're talking quite statically though. I don't think anyone is worried about today's models being a serious threat, but next year's, three years' time? Just three years' ago these models were useful bumbling fools and it's hard to judge where on the S-curve we currently are.
I'd rather be thinking about these issues in advance rather than waiting until the problem becomes real.
> I don't think anyone is worried about today's models being a serious threat
Fable is essentially bricked for my areas of interest (even being a member of the cybersecurity program). It seems like they’re attempting to sell regulatory capture under the guise of safety. That’s more of the point.
The models are still useful bumbling fools. We're in the flat part of the curve because we've exhausted existing data sources.
It's nice that people are genuinely curious about this.
- All of your observations are absolutely dead on
- Yet, we have very very very robust scaling laws that as Dario points out we've had and validated for over a decade. This extends to downstream measures like METR time horizon and compsosite benchmarks like the epoch capability index.
- If you look at where you're at now, which is again dead on, you're looking at a point on a curve that is quite easy to extrapolate, but less easy to tell when exactly on the curve a certain capability or use case undergoes a step change from error rates dropping below a threshold that is hard to anticipate in advance.
So while Dario / other frontier CEOs are understandably unpalatable, they are absolutely spot on with a call out that all of this is bound to happen and happen quickly, and that's without solving several core problems that haven't been solved yet (e.g. continual learning). In 2023, coding agents were just laughable. Yet they followed the same predictable training curves. Anyone looking at the data can see the obvious, and anyone reading newspaper headlines or hacker news comments would get a very different impression.
Are we plotting against cost? How is the capability advancement vs dollars paid for development?
By my read of the (very sparse) data, we're getting linear improvements in capability for super-linear increases in costs. [1] Indicates that by 2027 models will cost $1 billon to train. Dario estimates that model runs will cost $10 billion in 2026 [2]. That to me indicates costs are potentially growing faster than capability. Maybe by quite a bit.
If the value prop of LLMs doesn't prove out, that won't last. I'm of the opinion there is no data that shows actual economic value being delivered by models. The best data shows that LLM use might be destroying value [3].
[1] https://epoch.ai/publications/how-much-does-it-cost-to-train... [2] https://lexfridman.com/dario-amodei-transcript/ [3] https://unessays.substack.com/p/talk-is-cheap
I appreciate the data here but I don't think the read is quite right;
Saying we have linear capability for super-linear cost compares an unbounded variable (dollars) to bounded instruments (because benchmarks saturate). On unbounded measures, growth is exponential; you can see METR time horizons double every ~4-7 months (https://metr.org/blog/2026-1-29-time-horizon-1-1/). And capability being proportional to log(compute) is what the scaling law predicts.
Epoch puts training cost growth at ~2.4x/year as your link shows. Meanwhile cost for fixed capability falls ~10-40x/year (https://epoch.ai/data-insights/llm-inference-price-trends), and lab revenue is growing ~10x/year! Anthropic went from $1B to $9B to $30B+ run rate in ~15 months, OpenAI ~$25B.
On [3]: the "destroying value" conclusion flips sign on an assumed 15% baseline rework rate. The report's most direct metric is +16% merged PRs per dev. The RCT evidence is genuinely mixed (METR: -19%, with n = 20 and Claude 3.x; Cui et al: +26%) but its just super hard to do this well, I think Faros stuff was pretty cool, I haven't seen this before so thank you for the reference.
>"On unbounded measures, growth is exponential"
Maybe. There was a great comment in the thread on Fable 5 yesterday about benchmark comparisons between Fable and the latest opus models. here it is: https://news.ycombinator.com/item?id=48464600.
You could be right, but this is the most direct benchmark comparison I could find and it's not that strong.
>the "destroying value" conclusion flips sign on an assumed 15% baseline rework rate. The report's most direct metric is +16% merged PRs per dev.
I discuss this directly in my analysis. There's also an 860% code churn increase ratio. You only need 9% of that to be allocated to wasteful rework to drive throughput flat to the 15% rework baseline. Not to an assumed ideal state where there was no rework.
But even if it were not true, a 16% throughput improvement is pretty weak given the investment - especially given the direct evidence of quality degradation. IMO.
I appreciate you reading my stuff and taking the data seriously. Thank you.
Productivity != value.
Thanks for the story.
METR's time horizon is not a reliable metric of LLM capability growth: https://www.transformernews.ai/p/against-the-metr-graph-codi...
Yes I've seen this before, and while the critiques are fair and high quality (and unfortunately not unique to METR) we're missing the forest for the trees here.
First of all, if you take the articles critiques and work out the implications on the METR graph, all you're doing is shifting the curve up or down, it doesn't change the fact that progress is scaling exponentially. While it is technically possible the universe could be throwing a massive pathological curveball to change the conclusion from METR data (which is we've been seeing exponential growth over the last 6 years), I think that seems very far from likely. The fact that we see the same behavior from a variety of sources over a wide variety of tasks and domains is a pretty clear indication that METR while certainly far from perfect is actually painting a consistent picture at least in terms of the rate of progress.
You can look at ECI for a summary benchmark statistic, which does NOT use METR's benchmark, and you see a similar trend. Same with SWE-bench where the task distribution is far more in domain for real world problems. It is a bummer that this METR data can't be better funded. It would probably take $1M or so to really beef it up properly which any of these labs probably have in their couch cushions.
Wow. This deserves to be much more widely read. Thank you for this.
>By my read of the (very sparse) data, we're getting linear improvements in capability for super-linear increases in costs. [1] Indicates that by 2027 models will cost $1 billon to train. Dario estimates that model runs will cost $10 billion in 2026 [2]. That to me indicates costs are potentially growing faster than capability. Maybe by quite a bit.
This is true and well established.
As long as you get any improvement whatsoever, it is worth spending to train since it pays off during.
Imagine training was not $1 billion but $100 billion but the performance improved by just 10%. This is still worth it because you can squeeze out the profits across years and years right? The improvement is ever lasting.
> The best data shows that LLM use might be destroying value [3].
This is basically a conspiracy theory and if you really believed this, you should not have led with "How is the capability advancement vs dollars paid for development?" because if there were no value, it doesn't really matter how much you invest.
>This is basically a conspiracy theory
I think this is pretty uncharitable, especially when I've provided you with a dataset you can evaluate yourself and an argument you can review for logical inconsistency.
I have worked quite hard to locate data that supports your thesis, I can't find it. I've at least gone to the effort of documenting that search. Before you throw around such strong convictions, I suggest you actually look for yourself.
Respectfully, your link is not very convincing.
But what’s interesting is that you are commenting on a post where Dario is suggesting that LLMs are so extremely powerful that they can take over, help synthesise bioweapons, help in warfare, help in drug discovery — the whole post here is to try and regulate this. If you believe AI can’t even create positive value let alone discover new things then your problem is somewhere else and not in something like “but training costs a lot”.
So it is absolutely strange and contrasting to see you believe that LLMs are so weak as to create negative value while the CEO is asking about regulations because AI is too powerful.
I don’t think I can convince you that AI is actually that powerful.
But let me ask you something directly: if you believe what you believe, you should also acknowledge that AI doesn’t need regulations in the context Dario is proposing since obviously AI can’t do anything he predicts. Do you agree?
> So it is absolutely strange and contrasting to see you believe that LLMs are so weak as to create negative value while the CEO is asking about regulations because AI is too powerful.
You wouldn't ask a chemistry professor to write code. So just because LLMs create negative value for software development doesn't mean that they can't be helpful for bioweapons synthesis, especially considering the range of chemistry and biology sources Anthropic would have fed to its LLM that wouldn't be publicly accessible. The LLM doesn't even need to be particularly accurate so long as the amateur bioweapons researcher takes adequate precautions before following its instructions and does some background research beforehand.
This is a ridiculous stance to take. That LLMs are simultaneously negative value but can also help synthesise bioweapons. It’s the sort of stance you take when you already feel ideologically against AI. I don’t think it’s coherent.
It's more about information availability rather than intelligence. An LLM has had access to more information during its training period than you'd ever even come across over a hundred lifetimes. It has been trained on billions of books and articles across every single subject that exists on the planet. Can you imagine what real intelligence could do with all that information?
>This is a ridiculous stance to take.
I encourage you to be more curious. We're just talking. We would learn more from each other without these strong statements.
Is synthesizing bioweapons positive or negative value ?
>Respectfully, your link is not very convincing.
I'd love to understand why. This would be valuable feedback for me as I try to make my writing and exposition better. Also, if you have other data, that also would be valuable for me to know.
>if you believe what you believe, you should also acknowledge that AI doesn’t need regulations in the context Dario is proposing since obviously AI can’t do anything he predicts. Do you agree?
I think you misunderstand my beliefs. On net I think how we're using LLMs destroys value. That doesn't mean no one ever gets value from LLM use.
My particular point about trillion dollars is - the main place Anthropic, OpenAI, and - hilariously - SpaceX think they will drive value creation is in enterprise applications. In that domain I think the evidence is very convincingly negative. I'm certainly not the only person who thinks this. It's pretty well accepted in economics right now that there is no observed organizational level productivity improvement. Lines break down on whether it will show up eventually or whether we will wait forever.
My belief about LLM value is that it's most useful for individuals and small teams. Places where coordination and trust are easily established and feedback loops to value creation are tight. They are "short range" as it were.
Their value starts to erode as soon as a user becomes disconnected from the point of direct value creation. Which is pretty much everyone who works inside of a large organization. It becomes negative at pretty small scale, IMO. I do think there are patterns of use that could drive value at these scales. I talk about that in my post.
On Bioweapons in particular, I could see small teams of people working to build something very dangerous. Having spent my formative academic years in a biochemistry and microbiology lab though, I do think the danger is overstated. Papers are not know-how or equipment. There's a lot of tacit knowledge that can't get written down that is super hard to acquire.
But, I'd be happy for us to regulate AI for dangerous applications.
My question would be - why would Anthropic build something they so clearly think is dangerous? If they were really building something deserving of the valuation they have, why build applications like this?
To my eyes - it's super weird that a company would build something they think is dangerous and turn around and beg the governments of the world to stop them. That's really strange behavior from my perspective.
You have engaged in good faith discourse, thanks! I'll reply in a bit.
I went through your post in substack (I think that's what you were referring to).
> I'd love to understand why. This would be valuable feedback for me as I try to make my writing and exposition better. Also, if you have other data, that also would be valuable for me to know.
I think it comes down to few things
- you took a single report that agreed with your statistics, for the sake or argument lets say I buy it completely
- you suggest that net value is lost simply because there are more incidents. this is a big jump
- you say that historically different technological improvements may have had similar patterns but this specific one is different because AI is stochastic
So it all really rests on you finding one distinction with AI and then disagreeing with the past trends.
I agree AI is stochastic and I'll put it this way: it is a high variance bet but it pays off. This is a bit hard for people to understand -- its a tool that works sometimes really nicely and fails other times. Overall you are better off using it but you need to use it enough to reduce variance.
Let me ask this: if you are so sure this won't lead to enterprise level productivity, how do you think this will show in macro trends? Surely you must believe that the valuations must drop wouldn't you? Can you come up with a concrete future scenario that would vindicate your opinion that AI doesn't make enterprises more productive?
> My question would be - why would Anthropic build something they so clearly think is dangerous? If they were really building something deserving of the valuation they have, why build applications like this?
I think this is fair and interesting question. Here is what I think they think: If they don't build it, someone else might do it. And they think they are more moral than others. If they have a head start they can set the political and regulatory landscape.
>you took a single report that agreed with your statistics
These are not my statistics. I'm not affiliated with Faros at all. I built an analysis on top of their reporting.
And, it's also not one report. DORA has tracked statistics with respect to throughput and quality as well. Those indicators are flat for throughput and negative for quality. The throughput flatness is also supported by the shovelware data.
I discuss both of those lines in How I'm thinking: https://unessays.substack.com/p/how-im-thinking-about-the-va...
>you suggest that net value is lost simply because there are more incidents. this is a big jump
I don't think it's a big jump at all. Incidents and bugs drive rework. Rework has to be subtracted from throughput. Product throughput is the only thing people pay for.
This type of analysis is done all the time in manufacturing and devops. Here's a link for you: https://reworkcost.com/benchmarks. I'm not bringing novel intellectual ideas to the table here.
Faros reports a 16% throughput improvement on PRs. They also report an 860% code churn increase. If you assign only 9% of that increase to wasteful rework, then the absolute throughput improvement disappears. This is a very simple, straightforward analysis of the operations data reported by Faros.
> - you say that historically different technological improvements may have had similar patterns but this specific one is different because AI is stochastic
I'm saying LLMs are unreliable. I think we agree on that front, you say:
>I agree AI is stochastic and I'll put it this way: it is a high variance bet but it pays off.
What I'm disputing is the "pays off" statement. That statement is amenable to validation with data. In my view, the data is saying it doesn't pay off. I think it says that very clearly. Across distinct lines of evidence.
>if you are so sure this won't lead to enterprise level productivity, how do you think this will show in macro trends? Surely you must believe that the valuations must drop wouldn't you? Can you come up with a concrete future scenario that would vindicate your opinion that AI doesn't make enterprises more productive?
I think LLMs can deliver value in the enterprise. I think the way to do that is to use them as quality checks and not as primary authors of intellectual work - like writing code.
Unfortunately, this use case would not support the expected 2-10x productivity increases that current valuations depend on. I do expect a major market correction in the near future. It would not surprise me if OpenAI or Anthropic are acquired. I think we're at risk of that happening within the next 1-7 months.
What would invalidate my beliefs? 1. Actual micro or macroeconomic data indicating economic productivity is increasing. 2. A Faros like observational study demonstrating sustained throughput improvement with significantly less rework and quality impacts.
I think I could be swayed against the market correction if the financials of OpenAI or Anthropic are strong. I'm anticipating they will be quite bad. I think Mythos was very expensive to train and I think the improvements in capability are sublinear. The inference costs are incredibly high.
I also have ideas about how Anthropic and OpenAI are trying to change their business models into enterprise transformation plays. Similar to Palantir. But this comment is already long.
>If they don't build it, someone else might do it.
No other players in the market other than US tech companies have the capital or the technology to train the models of the power of Fable. The way the Chinese model builders are building their models is by distilling from US models. So Anthropic, by building Mythos with all this bio data, has created the possibility that other actors can distill their models and do harm with them. (Not to say the Chinese are seeking to build weapons, but actors with their models might).
Wait, let’s stick to your bet of market correction.
“In 7 months I think their combined valuation would be higher than it is today.”
I think this captures everything you and I believe in. Do you want to bet against me?
Everything else is fluff.
My prediction is that you might walk back on this bet, so try come up with some other macro scenario you anticipate.
If you can’t come up with an easy testable macro metric for your bold claims on AI, I think it is a weak move.
I can make the bet more in your favour - the valuation of Anthropic + OpenAI will be greater than 20% of what it is today in 1 year.
This is much stronger than your claim and I think you should agree that this is not possible if your claims on AI productivity are true.
I'm happy to make that bet. Just not for money. I don't gamble at all anywhere in my life.
But I'm happy to write something publicly like "Simianwords was right about this prediction and I was wrong". Also happy for you to suggest alternatives as well.
That’s interesting. I commented something about this elsewhere but to me part of the exponential argument that loses me though is that it can often seem like a way to distract from issues that already exist which we should be working to fix. Things like autonomous weapons or mass surveillance are already here and rather terrifying and I would hope that we would dedicate our time to fixing those rather than having industry leaders focus so much on hypotheticals. While I guess the hypothetical scenario could be so bad that we must focus on it, I imagine a world which can’t come up with a way to spread wealth more equally or prevent mass proliferation of surveillance technology through profit seeking behavior will not be able to handle a digital super intelligence. So I keep coming back to the question: why is all I hear these industry leaders talking about is the threat of extinction? Maybe it’s just news coverage but I would love to see a leading lab release research on the health effects of subaudible sound in datacenters or other immediately present issues which would build good will towards these further out concerns.
>why is all I hear these industry leaders talking about is the threat of extinction? . . . I would love to see a leading lab release research on the health effects of subaudible sound in datacenters
It is straightforward for industry leaders to avoid living near data centers, but there's no way for them to insulate themselves from the extinction threat -- no way short of somehow eliminating the danger for everybody, which seems quite hard to do. Since industry leaders are as self-centered as everyone else, the extinction threat is what they think about.
Also, you describe the extinction threat as "further out". A lot of us think there is already some small amount of AI extinction risk being incurred every day. I.e., we think the period of danger has already begun.
I see. I wonder how this works out in terms of risk/reward. I suppose if you take extinction as -infinite cost than it would be the only issue worth thinking about. Where I think this line of thinking gets challenging is when you need to take in terms of a counter factual. A lot of these were already risks prior to AI (bioweapons, nukes, etc) so what’s the marginal increase in probability as a result of AI I guess is the question which matters. I could get more around this way of framing it than saying that AI itself is the problem. It’s just the being more capable as a species increases risks. I think a lot of these pushback comes from the fact that it’s often the CEO who stands to gain huge by saying his tool is going to end the world so we need public buyin to supporting it. If instead it was just framed as “general technological advancement” is dangerous but potentially worthwhile I think more would be on board.
I'm doubtful of this idea that the reason the CEO of Anthropic says that AI could end the world is because he "stands to gain huge" by saying it.
If he is willing to lie to give his corporation some advantage, he couldn't come up with a lie that would sound less absurd and outlandish to the average decision-maker, who doesn't have much time to learn about this particular technology?
It is more likely in my eyes that he says it because he genuinely worries about AI extinction risk -- like many people do who've studied the technology for a long time.
> this exponential line of thinking
It's a clever argument because if you question it, you're reminded of the entire history of technological development which is, guess what, exponential.
You're sometimes also dismissed as not understanding the concept of exponentials. This again is clever, as it's baked into the definition that if you don't see it happening, or can't imagine it happening, well that's precisely a tell you're living through an exponential!
All the reasons you might give can be countered with, essentially, "that problem that seems clear today will go away sooner than you can imagine and when it does you'll be on the back foot, so you'd better just assume it will go away and project/plan accordingly".
The trick is entirely that one cannot possibly deny the general power of exponential progress across all of technology, it's almost a law, but it doesn't work in the other direction - no particular local technology is owed exponential growth because of this general pattern. Sometimes things just cap out at merely 'useful' and don't improve much further, no matter how much you want to believe they won't, no matter how steep the progress curve (or, indeed, line) has been up to that point.
To this point the narrative of what these tools can do over these last 3 or 4 years has always been way ahead of the reality. Everyone who works with the tools knows this.
Not everyone wants it to be true, so some will not acknowledge it and will just keep pushing this year-ahead projection as ground truth today. Many (not all) of those people aren't builders, so they don't have to deal with present reality jarring up against this projection of what ought to be possible, they're safe just talking about what should hypothetically be possible and making plans around that that won't be executed for months to years anyway. This keeps the flywheel going, and in fairness, some of the reality has actually caught up in certain ways, so some of those plans will have to some degree worked out which spins the flywheel faster still.
In the end though I just keep thinking: it's been 4 years (as referenced in the post). A lot has happened, the tools are very cool and very useful for certain things. But when I put my head up and look around in the world, even just the software world, nothing's really changed in terms of actual outcomes, in terms of new things appearing or being built that didn't exist 4 years ago. Certainly nothing feels instinctively like it's improved much, subjectively.
Maybe this is what it feels like to be in the knee of a curve of an exponential, but it seems equally reasonable this is just a breakthrough that's kind of improving at a clip you'd expect it to for all the investment put in, but fundamentally is just a new tool that needs to be slowly commercialised in an economically rational way, as we gear up for the next breakthrough which may or may not be related. Who says it must just keep improving forever? This argument never made much sense to me.
Given that according to some we can code at 10x speed for at least half a year, I wonder where there are some autocoded softwares with 5 years' worth of equivalent human engineering work.
I've experienced the same.
That said Claude Code has a million features like loops that I know exist but never use.
I imagine that spending a lot more time creating an initial spec goes a long way towards independence, I just don't usually do that.
This is a very tech-focused message board, populated by mostly tech-insiders, so perhaps a little outside perspective will help people understand.
Tech people are following a religious belief system whose utopian promise is the all-powerful computer that will end all suffering. I once read an article in reason magazine from over 30 years ago about how an advanced computer in the future will bring everyone who has ever lived back from the deat and let them live in paradise. They were completely serious. Atheists reading this may object to my description of the tech belief system as religious, but I believe it is accuarte. The idea that tech is an imrpovement and will improve people's lives is believed as an act of faith. Tech has its own moral systems based on some form of libertarian progressivism. And in the future, through the inevitable scientific magic of exponential something, a computer will ascend to godhood and judge all mankind for their actions before allowing some into eternal paradise.
To what extent any of this is true is up for debate, but most west coast tech elite are actively working towards this future, and these are the ideas that drive them. It's hard to talk to them about it because this is their woldview, and they imagine everyone to believe what they do.
Heard a tongue-in-cheek comment about "building a god" from someone at one of these AI labs.
The builders believe that the machine you describe will judge them positively, purely because they are building the system according to their judgment and beliefs.
> Tech people are following a religious belief system whose utopian promise is the all-powerful computer that will end all suffering.
Uh, I don't really think that's anywhere close to an accurate characterization of most people here. Everyone, including Dario and any researcher at any frontier lab, knows the situation is quite scary and unprecedented. There are problems that will be solved and diseases that will be cured, but will we be living in an Orwellian universe? Will a rogue drone swarm find you cowing in your basement and murder you? I mean the technology for this is already mostly here, it's a matter of the willpower and budget to roll out something really evil.
The comment's question is about capabilities and why the discussion about capabilities often times is far removed from todays capabilities.
I dunno, I don't think an outside observer would be too hard-pressed to find the fervor with which some talk about the endless possibilities and miraculous works of AI to border on the cusp of religious.
Some people sure, but some people believe the earth is flat. What’s without a doubt is the impact AI has already had, something maybe 2 years ago was dismissed as religious fervor maybe. Statistics and robust measurements plus stable scaling laws make this pretty far from religious I would say.
Idk, I don’t see the same mindset in China. There’s a post floating around here about someone who visited the DeepSeek lab and said they just view it as another technology. Like a commodity. So I tend to think either the big tech ceo’s in America are delusional or malicious, or perhaps 50/50.
The role of the modern American CEO is basically to be a huge grifter, no wonder when I see Amodei or Altman they always remind me Sam Bankman-Fried, who btw has now applied for pardon from Trump.
I mean Amodei was hawking the doubling of human lifespan to some boomer investors, I wonder why he chose that audience. They are just afraid of dying like the rest of us plebs but instead of believing in god or other deity, they have turned the AI into one.
> A wide range of pro-employment policy incentives can help to slow or reduce job displacement, including: wage insurance policies that compensate people when they have to take a lower-paying job, retention tax incentives to encourage employers not to make layoffs, workforce training grants, or infrastructure to facilitate matching of employers to employees to speed the rate of labor market adaptation. While the particulars of which interventions are best will depend on what kind of labor displacement AI brings, we should readily accept the costs and market inefficiencies that these policies could entail, particularly as they are likely to be offset by AI-driven productivity gains.
People get income from one of three places: capital income, labor income, or the welfare state. If this technology truly unlocks a holy panacea of productivity with a commensurate drop in employment then capital’s share of the national income can and should provide for a wider and deeper welfare state. Nothing new need be invented here. Dario’s long and only somewhat organized list of policy interventions makes appropriate preparedness sound like a manic pulling of any and all levers when a simple theory of distribution will suffice.
This and we already did a dry run of ad-hoc distributions with COVID relief. They had to use the data from tax filings but it did work in terms of getting the money out there.
> If this technology truly unlocks a holy panacea of productivity with a commensurate drop in employment then capital’s share of the national income can and should provide for a wider and deeper welfare state.
This isn't guaranteed in the tax system as it exists today, because reinvestments into further growth are often treated as expenses which cancel out the income for tax purposes.
You’re conflating firm level taxable income with the national income.
No I'm not? Current American tax policy does not guarantee that any fixed percentage of the national income will be received by the government as revenue. If the advent of powerful AI pushes corporations away from dividends and buybacks towards expansion and research, then tax revenues may flatten or even decrease even as the national income spikes. (Sales taxes are more likely to track aggregate economic activity, but US sales taxes are both not very high and don't flow to the federal government.)
It's crazy how all these tech CEOs develop the same sense of ethics that seeks to make the foundation of open research and development that made their efforts possible and may threaten their market position illegal in the name of safety against nebulously-defined risks
It is impressive how well they've scheduled all their releases, posts, and other news to dominate the tech news cycle almost every day in this pre-IPO phase.
Does this guy have exponential breakfasts?
I agree on some points about the missuse of AI particularly for surveillance, military and propaganda.
But this reads like a post further glazing Mythos, and we are just one or two years away "trust us guys", and similar to Mistral's policy plea "please use AI everywhere or we are going to be left behind".
I had the hardest time accepting one of his first points that LLMs could barely write a line of code 4 years ago.
ChatGPT 3.5 was reasonable at code writing but hallucinated a lot of library functions. Yes we have better harnessing today, and models have been further finetunned with reallife code, but pushing this argument just to support his exponential narrative is deceptive. Like most AI marketing.
While I do understand the risks, I don't understand the solution. Essentially, Dario is saying that powerful model weights can't be distributed (ban open weights), and governments should coordinate and agree on standards, and block any dangerous model from being used at all, with government deciding what dangerous means.
Okay, I don't understand how legitimate access is granted then. Surely, Dario isn't saying to ban Sonnet, because I can definitely make it do cyber harm, as most exploits that I've seen in the wild with my own eyes were trivial.
So the only way I see his proposal working is:
- No open weights, AI is centralized in the hands of few
- We get AI-FAA that sets the rules and monitors
- If I want to do a security scan of my codebase, I get a time and scope limited license from AI-FAA that I upload to claude that will allow it to run the security scan in cloud with their models - Claude Mythos Scanner(TM).
Dario's proposal ultimately requires that people lose direct access to inference via API. Is this why they've been building SaaS clones with AI bolted on?
> AI models have gone from barely being able to write a coherent line of code to writing most of the code at major AI companies.
Gasoline has gone from barely being able to power stationary farm machines to now being the fuel that underpins our entire economy. So, great news all around, right?
> which predict an exponential increase
And was that actually delivered?
Real question: If a model goes from 80% accurate to 85% accurate is that an exponential increase in "cognitive capabilities?" Are we considering training costs and effort?
Yeah I’m lost by the constant use of “exponential”. What is the x and y axis when the talking heads say this?
I think people are typically referring to the task-completion time horizon at a fixed success rate [1]. That has had pretty robust exponential scaling for many years now.
[1] https://metr.org/time-horizons/
I read this essay, and it feels like lying behind a mask of moral responsibility and safety for humanity.
They are asking for FAA style preclearance and third party audits. That literally means no new AI startup can emerge. Do they not know that audits cost money?
Protect your own monopoly, protect your customers' regulations. They want strong regulation like the FAA to raise barriers to entry for the foundation models they themselves build, but then why do they want to loosen FDA regulations? While at the same time driving token consumption from their own customers.
They talk about permanent job displacement and UBI. I usually call this "a morally packaged safe landing."
They are doing something unpopular (destroying jobs) and getting criticized for it. But they do not want to be criticized further, and they want to ask for social sympathy. So they claim a 'noble cause' that everyone can sympathize with and that is safe for themselves
AI will generate astronomical productivity gains and capital profits, which AI companies privatize. So why should the social costs be paid by national taxes? In my opinion, something like "We will donate all of our AI companies' revenue for the next 10 years to society" would show genuine sincerity.
Then they say, if we do not develop AI, China will eat our lunch, and they go after China. But is not this really about preventing Chinese dumping, maintaining our own token prices, and asking the world to beat down China so that they can preserve global tech hegemony?
But by blocking China from the CUDA ecosystem, now the CANN ecosystem has emerged, has it not? If China develops techniques that reliably reduce inference costs, who knows how things will turn out then.
Honestly, I like Anthrpic's Claude, but the Anthropic CEO's rhetoric is so stale. It is not that it feels hypocritical. It is that this is just a one dimensional rhetorical tactic that assumes the public is stupid.
I do not think open source is unconditionally good. (It is good, but it can become bad in all situations or all countrie). Open source itself is a barrier for countries outside the Anglosphere when they want to release IT products. Because there is no incentive to buy a product that is worse than an open source alternative. So I do not think everything necessarily has to be open source.
But this (referring to Anthropic's position) seems to treat people like fools. If regulation is needed, shouldn't they also argue that FDA regulation is needed? I wish they would be consistent
> They are asking for FAA style preclearance and third party audits. That literally means no new AI startup can emerge. Do they not know that audits cost money?
Training frontier AI models costs money, orders of magnitude more than third-party audits. If you can afford to build the model, you can afford to have it audited.
Cost as 'money' is not just about a one time audit fee. What I mean by money is that pre approval also affects time and human resources. In reality, it is difficult for startups to hire people to handle this. For example, DeepSeek in China has 300 employees. If they were subject to regulation, they would typically need to hire about 5 to 10 additional people for a compliance team. That is hard for a small company. I partially agree, but even if the monetary amount is small, the very fact that fixed costs arise in organizational operations is the real problem
As per usual in situations like these, one must look at the actions in order to assess whether there's any worth in the words. And the actions of Anthropic have, by and large, been steering hard towards establishing a walled garden, empowering corporations over consumers, pushing for regulatory capture under the guise of national security, and consolidating as much power as possible within Anthropic and no one else.
He is certainly skilled at writing philosophical essays that sound like they make cogent and thoughtful points (and sometimes genuinely do make cogent and thoughtful points), but his company's actions disregard his rhetoric at their best and actively contradict it at their worst. For instance: there was zero pressure on Anthropic to release this model to anyone - they were ostensibly in the lead, which is the exact scenario they said they'd hold back model releases back when they axed their safety policy the instant it came under the slightest amount of economic pressure:
> And it promises to “delay” Anthropic’s AI development if leaders both consider Anthropic to be leader of the AI race and think the risks of catastrophe to be significant. https://time.com/7380854/exclusive-anthropic-drops-flagship-...
Yet this essay proposes this extreme auditing and regulatory administration pipeline that new models are supposed to go through before they release, right after they, themselves, under no pressure, ran a months-long marketing campaign under apocalyptic rhetoric, which they continue to harp on to the point of nerfing/auto-downgrading their model into uselessness for many legitimate tasks that older models had absolutely no issue supporting, while the supposedly extremely dangerous version... can be freely used with no guardrails by their corporate partners.
The hypocrisy here is neither difficult to see nor is it particularly sophisticated, which makes it all the more infuriating.
It's very hard those days to think of companies/people more arrogant than Anthropic/Dario, which is quite the achievement as the bar is very high.
If that arrogance was well placed at least you could somewhat excuse it, but the fact that it is so overtly hypocritical and based on false premises just makes it so much worse.
>The government should have the power to block or deter deployment of the model if it is determined, in light of third-party assessment, to present unacceptable risks. This power must be scoped to the above four specific risks and there must be protective measures against political favoritism or arbitrary decisions.
I feel significantly less sympathy for Anthropic's Supply Chain Risk designation if they believe the government should have this power over them. You get what you sign up for.
These last few days, I can't help but think that we're now at crossroads that future people will remember as one of two:
- And this were the first steps of Anthropic establishing worldwide corporate technocracy.
- And this is when Anthropic lost and everyone got access to AI.
Similar to how IBM's defeat allowed us to have PCs.
Regarding regulation: I'm deeply invested in computer vision systems and i fear that policymakers [who are not deeply familiar with the technical distinctions between AI systems] may write broad rules that cover "AI" generally. In that case, computer vision companies and industrial users could end up subject to requirements that were largely motivated by concerns about generative AI and LLMs.
What will be Amodei's job after we have AIs that are better at evrything than humans? Is the AI going to care about our stock exchange playgrounds that reward the future Antropic stock holders?
> after we have AIs that are better at evrything than humans.
That this is worded so definitively is a testament to the success of the AI industry. The idea that LLMs will be "better at evrything than humans"[sic] is far from certain.
I suspect that if someone does invent a machine like this, it won't look like a 2026 LLM, and it will be far far in the future. everybody relax.
“Hey everyone, no need to worry, bigfishrunning suspects it’ll be fine!”
I'm glad someone agrees! in all seriousness, there's really nothing you can do about it, so why worry? If the AI monster is gonna get us, then you might as well enjoy your life until that point.
Well obviously he'll have the AI 'dispose' of the poor and live a life as a king with a select few farmed humans and have the world as a play thing.
Really the entire future of AI at this point seems like "Don't worry about it, we'll figure out when we get there". Works a lot better if you're extremely rich and can afford your own private security.
I never understood why people setting up bunkers expect the security to still be loyal after whatever happens.
I'll sell them unquestioningly loyal security LLMs if they want.
Prompt: The robber is stealing the crown jewels.
Output: You must stop the robber from stealing the crown jewels.
Even loyal security would hardly be sufficient. If a million people have decided they want your head on a pike, even a billionaire cannot afford a big enough army. And what exactly is a billionaire in that world anyway?
Why are some people loyal to corrupt governments?
What happened to indira ghandi?
Indira Gandhi tried religious divide and rule way too many times - in Kashmir and West Bengal/Assam she got away with it while she was alive and it blew up later but Punjab was her Antietam.
The point being she got gunned down by people who she thought was loyal.
Once a human goes after a class of people - the people who protect such persons’ also take them out.
Can't find the article mentioning it but apparently it's an open problem they're thinking about.
But yeah if society collapses these billionaire nerds are the first to go. Quietly, in their bunkers, while the team leader of their seal mercenary team takes over.
Even before the rest of us realizes what's happening.
Found it
https://www.theguardian.com/news/2022/sep/04/super-rich-prep...
> Finally, the CEO of a brokerage house explained that he had nearly completed building his own underground bunker system, and asked: “How do I maintain authority over my security force after the event?” The event. That was their euphemism for the environmental collapse, social unrest, nuclear explosion, solar storm, unstoppable virus, or malicious computer hack that takes everything down.
This single question occupied us for the rest of the hour. They knew armed guards would be required to protect their compounds from raiders as well as angry mobs. One had already secured a dozen Navy Seals to make their way to his compound if he gave them the right cue. But how would he pay the guards once even his crypto was worthless? What would stop the guards from eventually choosing their own leader?
The billionaires considered using special combination locks on the food supply that only they knew. Or making guards wear disciplinary collars of some kind in return for their survival. Or maybe building robots to serve as guards and workers – if that technology could be developed “in time”.
I tried to reason with them. I made pro-social arguments for partnership and solidarity as the best approaches to our collective, long-term challenges. The way to get your guards to exhibit loyalty in the future was to treat them like friends right now, I explained. Don’t just invest in ammo and electric fences, invest in people and relationships. They rolled their eyes at what must have sounded to them like hippy philosophy.
>AI Exponential
How much of the policy prescription changes if the exponential is actually just a series of sigmoids[1]?
[1]: https://x.com/ylecun/status/1799064075487572133
Ooof a Yann LeCun quote from '24, a risky time period to be mining his quotes from lol. But I think broadly he's right, but the argument is: we don't see any evidence for the curve flattening, and we should plan for it _not_ to flatten any time soon. "It will eventually slow down" is true but its meaningless if you dont put any sort of time period on this.
I have to be honest, I am tired of reading these arrogant, self-absorbed posts from Dario and Anthropic in general. Opening with this lord of the rings reference just feels like they are trying too hard and are untrustworthy.
As annoying as their tone is, the real big danger is what they are setting up for. All this fear-mongering around Mythos, the overly aggressive controls on Fable, and these manifestos they keep writing, are part of setting up for REGULATORY CAPTURE. Even collaborating with the Pope and the Interfaith Alliance (https://iafsc.org/our-work/faith-ai-covenant) are part of creating a vast support network for regulations and restrictions. Those regulations will help those faith organizations or the government or whatever, but will also help Anthropic’s bottom line.
Those regulations will not support your civil liberties. They will restrict speech, access to AI, and allowed uses of AI. They will lead to bans on use of models from some countries like China, and also bans on open-source or open-weight models.
If Dario wants to be trusted, he needs to explicitly say in writing that Anthropic will not support any legal or regulatory restrictions on open-source AI, open-weight AI, or Chinese models. Otherwise, what he is really saying - even as he claims he is trying to ‘defend democracy’ - is that he and Anthropic do not truly support fundamental rights like our right to speech.
It’s not just Anthropic either. OpenAI had their own recent polemic, pushing for regulations like mandatory safety reviews by agencies for “frontier” models (https://news.ycombinator.com/item?id=48387246). It’s a dead giveaway that these companies have no moats, are in serious danger of being a commodity, and are now in the process of using regulations and enshittification to hold onto money and power.
It’s easy to dunk on them because of incentives, but consider for a moment a world in which they have access to internal, non-lobotomized models, have run evals on them and have been genuinely concerned by what they saw.
but consider for a moment a world in which they have access to internal, non-lobotomized models, have run evals on them and have been genuinely concerned by what they saw.
That world arrived with GPT-2: https://techcrunch.com/2019/02/17/openai-text-generator-dang... , https://www.weforum.org/stories/2019/02/amazing-new-ai-churn...
These people have been distinctly unreliable at predicting the robot apocalypse, yet they demand a degree of control that shouldn't be entrusted to a genuine psychic oracle.
If that's the case, why not publish that research?
Until they can show receipts, we're forced into a binary situation of "Do you trust the CEO of a lab with a trillion dollar valuation quickly approaching their IPO?"
Maybe he's right, but from an outside perspective it just looks like an attempt at regulatory capture to pull the ladder up behind them.
I mean he wants to be trusted but he also absolutely doesn't support freedom of speech in that sense, and I doubt anyone with power or influence over ai policy does?
If you find a good lobbying group with money who can push for it let HN know.
To be honest, at this point I’d take any group at all willing to usefully and non-hypocritically push for free speech (as a general societal value, not just a US-specific legal notion).
https://en.wikipedia.org/wiki/Foundation_for_Individual_Righ... has been doing alright recently?
This was slightly more thoughtful than I feared it would be. However, what I still do not see in any posts from any of these AI pushers is any genuine consideration of the possibility that the best thing might be for them to do less of what they are currently doing. Not to do it differently. Not to mitigate it. Not to do something else in addition. But to actually reduce their current activities.
The only effective action is push back on Athropic.
> A nation that possesses powerful AI facing one without it—or even facing one that is behind in AI by 3 years—could be the equivalent of an army of World War II Marines facing an army of medieval swordsmen.
This is a somewhat ironic take from someone who very publicly feuded with the US government about whether their AI could be used for waging war.
His entire "I Have No Moat And I Must Scream" essay rests on the assumption that our government will remain aligned with both his own company's interests and those of our democratic society as a whole... as if our government is even starting out from such a position. After his recent head-butting defeat by Hegseth's Department of War (sic), it's just downright surreal to read passages like this:
Either he's playing us all for fools, or he's playing himself. I suppose both could be true.> As a company, Anthropic always does as much as it can to work with customers to find creative new use cases and new sources of revenue that allow them to do more with their existing workforce, rather than focusing solely on cost savings (which often means reducing the workforce).
Without direct workforce or policymaker representation on the boards of private entities, the private sector will seek to maximize shareholder value even if that means workforce reductions.
It's not clear that any country could realistically ensure that incredibly powerful industries/private sector entities operate perfectly aligned with national interests, short of nationalization.
Large tech companies are already quasi-state actors. In theory, international law and regulations can be binding and enforceable. We see how well that works in practice.
> Members of the trusted coalition should freely share chips and semiconductor manufacturing equipment (SME) with each other, while working together to deny it to adversaries. US export controls on frontier chips and SME to China have been a major contributor to the US’s overall lead in AI, and these policies need to be expanded, tightened, and coordinated with other likeminded states.
I understand why Dario thinks this is crucial, but it's a very dystopian view of the medium-term future.
I'm not an optimist to the point that I believe that AI will lead to global Star Trek-style utopia (although it theoretically could), but ongoing disparity between "allied" and "enemy" powers relating to hardware technology and software models is both not really possible to enforce in the long term, and a pretty dismal state of global affairs even if successful.
I'd be interested in an expert geopolitical opinion on what the long tail of this would really look like in any sort of reasonable reality.
This guy can’t stay a day without posting something more or less “ban open source AI”. We keep you safe
"AI is advancing at a lightning pace—in only four years, AI models have gone from barely being able to write a coherent line of code to writing most of the code at major AI companies. Similar gains have been made in biology, physics, math, finance, law, translation, and many other fields."
This is a massive exaggeration. The advancement in the automation of computer code writing has been impressive and is obviously, at least in the short term, changing the software engineering industry substantially. Most other fields have not been affected to nearly the same degree. Certainly not biology, physics, finance, and law (I don't know enough about the math and translation fields to speak to those).
---
"3. Accelerating AI’s positive impact..."
This whole section is the type of thing that often comes out of the mouths of Silicon Valley tech executives without a pharma background. It indicates a thorough lack of understanding of the realities of pharmaceutical research. What he is describing here is removing many of the solid, evidentiary rules that are in place to make sure that the drugs reaching the market actually work and replacing them with proxy predictions. Look, my least favourite part of the job is the animal testing, and I would be hugely grateful if that could be eliminated from the drug discovery pipeline. People have been trying to do that for a long time. But it's extremely difficult. Biology is very, very complicated. Our understanding of how processes in organisms work are vague and approximative. This is not computer code. Even if Anthropic somehow got all of Big Pharma to hand them their proprietary data, it would only scratch the surface of the understanding that is needed to solve these kind of problems. Due to these realities, the program Amodei is describing here would, effectively, open a floodgate of drugs on the market that don't actually do what they are supposed to and are more likely to have unidentified toxicity.
Also I’m not denying it would have a major impact either way. But c suite have basically demanded people use ai to code. It’s not all organic
>> Of all tyrannies, a tyranny sincerely exercised for the good of its victims may be the most oppressive. It would be better to live under robber barons than under omnipotent moral busybodies. The robber baron’s cruelty may sometimes sleep, his cupidity may at some point be satiated; but those who torment us for our own good will torment us without end for they do so with the approval of their own conscience. They may be more likely to go to Heaven yet at the same time likelier to make a Hell of earth.
-C.S. Lewis
> Often this is for good reasons: governments have grave powers, and it’s usually for the best that they aren’t used too hastily.
"Don't mention the (Iran) War!"
How predictable. The company currently on top wishes to use the regulatory power of the state to prevent competitors from encroaching on their market dominance. It’s a tail as old as time, although their CEO’s rarely publish blog posts about it.
Yep
> "Models above a threshold of compute should undergo mandatory testing by a qualified third party for their level of risk in four specific areas: cybersecurity, biological weapons, loss of control of AI systems, and automated R&D that could accelerate these other risks."
AKA: Make it as expensive and untenable as possible for any open source model to jump through the regulation red tape so we can pull the ladder up behind us.
Disgusting.
All the marketing talk about "this model is so dangerous" "omg we can't release this to the public its so dangerous" etc. is just priming for the incoming lobbying for protectionism from foreign competition, and regulation preventing the development of any other model that could threaten their dominance in the name of "safety"
Yeah, they know their moat is evaporating as the Chinese models continue to catch up, and at a much cheaper price.
Also, why has my comment been flagged? It is sitting at positive votes but has suddenly been flagged for no apparent reason?
If you mean https://news.ycombinator.com/item?id=48481131, users flagged it. We can only guess why users flag things. Perhaps it was because they thought the comment broke the site guidelines by being snarky and/or fulminatey?
I don't think it was an extreme case of that, so I've turned off the flags now.
Thanks for staying on top of these things dang - such an important topic so it’s great to see the discussion rage but stay constructive.
Thanks big dog
I mean it’s pretty tame and actually adds to the conversation.
It feels somewhat obvious that someone just disagreed and flagged it.
Is there a system for holding people accountable who abuse flags?
Yup there is.
I've restored a few other comments that were similarly flagged as well.
Dario has been riding this exponential for longer than almost anyone here, I’d recommend people try to not scream ‘regulatory capture’ immediately when the risks have indeed materialized and the trend critically does not show any signs of slowing down, in fact the only disagreement is whether it’s accelerating. You have to start thinking in log scales to be able to forecast anything.
Of all the points, I find only this one fair: Dario is making it hard for competitor startups to come up because he's proposing additional regulations.
A good proposal here is: should Anthropic and OpenAI become sort of VC's that fund other competitors?
With what money you fool?
They are deep in the red. That’s before considering reinvestment needs.
If you think they are deep in the red why do you want any regulation? They will cease to exist. You don’t have to worry in that case.
You must be pretty stupid to write such a post.
Regulation puts up a barrier to entry, enabling incumbents to have greater market power.
[flagged]
This reads like an AI with an overflowing context window wrote it; or in the alternative it’s a list of statutes written by an arrogant and delusional king. It is this type of arrogance that will lead to an unfavorable reaction by Congress.
The sooner the better.
https://xcancel.com/cuimao/status/2058458683781365873
Honest question: is there a reason for the naming conventions for these models? Anything that makes it better than giving them names with model numbers, like “Claude 3” or such?
Yes.
Each of the nouns is a “size class” in literature. From small lines poem (haiku, sonnet) to larger story (fable) to very large story (opus) to culture-defining foundational (myth).
It’s a fun way to say how many parameters are in the model without revealing a number like 405B or 17B which isn’t really comparable vs other models.
Interesting. Thanks for enlightening.
My non-poetic brain thinks we should call then Mini, nothing, Pro, Max, and then version numbers. Exactly like Apple. It'd be so much easier to parse. Maybe the AI companies like having the affectionate names haha
I personally hate the Mini, Nothing, Pro and Max nomenclature. I find it very tacky and the confusing having to remember a "base model" exists without any descriptor.
Apple does make it worse though, they have products like the iPhone Pro Max, which combines two!
Oh yes, let's induce regulatory capture and ban open-weight models in the name of alignment.
When has giving power to politicians have had a good result? Please don’t
I feel completely baffled by the other responses on this thread. People viewing this purely as a marketing stunt, regulatory capture or attack on their freedoms, with seemingly no appreciation of the real threat that AI could pose to society and even humanity given its current rate of progress.
I'm not going to claim that the CEO of pre-IPO company has no incentive to bolster the claims of his tech, but to completely disregard everything he is saying based on that seems awfully binary.
I don't know whether people are just high on copium, spouting "it's just fancy autocomplete" or "only humans can really be creative" on every LLM-related thread, but it is impossible to deny that in a span of a few years we've gone from models that could barely put together a sentence, to something maybe not equivalent to a junior developer, but at least resembling it.
And sure, you can point out every flaw that current day LLMs have, just how everyone pointed out that Stable Diffusion couldn't generate accurate hands (until it could 6 months later!). But the gradient is pretty clear and I am yet to see a well-argued narrative from anyone why scaling laws should fail in the next year or two (by which point it feels like we're going to have a real problem, extrapolating the current trajectory).
I'm very glad this discussion is at least being had, and I wish everyone would get off their high-horse and take things a bit more seriously.
> I'm not going to claim that the CEO of pre-IPO company has no incentive to bolster the claims of his tech
I am going to say it. The CEO of a pre-IPO company has extreme incentive to bolster the tech he is selling, to the point where his every action should be viewed as only in service to that goal. Every word he says should be viewed critically through that lense. He is not making this post out of the goodness of his heart, he is doing it in service to the IPO. If it happens to align with your views that's great, but it's still just a marketing stunt to get people with your views to buy in. Don't be fooled. Buy in if you feel it's a good deal, not because of the CEO's marketing.
In terms of aeronautics, went from the Wright brothers to the moon in 40 years. After that, everyone understandably thought that we'd be living in space and flying everywhere with personal jet packs in another couple years. Little did they know, it was the top of the S-curve, not the middle.
In the 60 years since, we've barely been able to adapt the 737 to fly longer routes.
Sure, I have no doubt that AI progress will follow an S-curve. The question is, where are we on it and is the plateau at a level safe for humanity? That's a very difficult thing to estimate without a crystal ball and not a risk I want to take.
Or simply we use AI and see on the ground what it can and can't do. I can generally trust an agent for solved problems, but the more something deviates from established industry standards (i.e. what was relentlessly scraped) I have an increasingly harder time not having constant oversight of what it's doing, no matter the specs I put on the md.
Personally I feel most of the improvement in the last year comes from tooling/integration (MCPs, realtime documentation access, treesitter support, orchestration) than from the models themselves, in the last year. And still frontier models would routinely come up with bs until you tell them to actually use those tools.
You're talking as if this is a static thing though. It's the God of Gaps [1] but for humanity's special sauce.
Two years ago, I couldn't trust an LLM to do anything that wasn't straight forward boiler plate.
One year ago, I was pretty solid at writing algorithms that were combinations of existing ideas.
Now, Fable is outputting stuff that I would genuinely consider to be creative and original if a colleague had presented it to me.
Yes, maybe the code style still isn't great, but given the pattern of the last few years, it feels correct (a priori) to assume that this gap isn't going to keep closing.
[1] https://en.wikipedia.org/wiki/God_of_the_gaps
I agree that AI poses a threat to society. I act on this by not developing world-leading AI models and offering them to anyone willing to pay top dollar, while funneling that money back into accelerating AI capabilities development. Maybe Dario would consider taking a similar ethical position? Maybe he would support restrictions and taxes on data center construction, in order to slow down the pace?
If Anthropic were not developing these models, one of many other companies would be. I think it's good that the CEO of the current world-leader is at least considering these discussions and platforming possible solutions.
The fact that he doesn't support more restrictive approaches that don't align with his incentives doesn't invalidate the points he is making.
> If Anthropic were not developing these models, one of many other companies would be.
"Everyone else is doing it" but even worse as nobody else is actually even doing it and you just made up that part.
Just commenting on this horrible justification trick in general.
It’s a guy asking to make an end-run around the constitution and the APA regulatory framework based on a flimsy sci fi premise. Naturally it provokes a negative reaction.
Why are they not preaching for protected weights, but public, ie, under state control? What do you feel his posture will be if that is starting to be discussed?
Also, on an unrelated note, why would you have an account for 5 years and only now post your second comment? AI has been an existential threat for years, why only now?
This is a pattern I am seeing all over the place on HN in the last year in AI threads, and I have to admit that I am starting to become paranoid and my feels need some assuaging.
Your first point is very reasonable, and I agree that that is something Dario would likely be more opposed to.
However, my point isn't that I think Dario is our saviour who we should follow the every word of. As with everyone, his opinions should be filtered through the lens of his incentives. That said, I don't understand the knee-jerk reaction by many commenters to completely disregard the many important points he's making.
As for the lack of my account use, I can't comment for others, but I'm just quite shy. I've opened up the comment box many times to write a reply but rarely commit to actually posting it, especially because I feel like I'm not on the side of the general HN consensus.
You write well & HN is better when there are more well-written people on the opposite side of the consensus
Genuinely, thank you. This is very encouraging and makes me feel much better about commenting more going forwards.
The quality of discussion and prose on HN is just generally so high that it can feel quite a bit intimidating to jump in (in contrast to Reddit where I have no worries about commenting haha).
This 100%. It’s great to hear diverse perspectives!
Here's the real reason you are finding these responses in the thread and I'll lay it out in the open.
There are people who simultaneously are scared about AI but refuse to believe that AI is the * real deal * and can do everything Dario thinks it can. These are the same people who think its all marketing hype and Dario is "hyping up" before IPO or some lame conspiracy theory.
It is high time people start accepting the real world performance of LLMs and brace themselves instead of hiding behind two contradictory views
1. AI is hype
2. AI is scary
Those two views aren't contradictory. AI can both be scary, in that the primary dangers are bias, misinformation at scale, and displacement of human labor, and simultaneously be overhyped in the way it is marketed at sold to investors with the "what if this happens in the future" scare mongering.
Notice how in any of these proposed regulations, Dario is talking about future advances. Notice how these suggestions are never implied to apply retroactively to existing models. If AI was SO dangerous, then any future regulations should obviously be retroactive and we should seriously consider restricting access now to the models that already exist.
Hard not to see that as nothing more than a play at regulatory capture and pulling the ladder up behind them.
Humans just aren't very good at dealing with threats that aren't immediate concerns. 'Safety regulations are written in blood' is a saying for a reason. A significant chunk of the population shrugs off climate change, and nearly all fertility rate crises threads are filled with dumb 'hurr lower population good' and/or 'See what Capitalism gets you!' rhetoric - They fundamentally don't even understand what the problem is. So is it really all that surprising that a technology like this would be shrugged off until it's too late ? Especially one with such existential issues for humanity? Some people are still too loathe to admit the clearly intelligent machine is intelligent, devolving into increasingly nonsensical and absurd (and ironically more human demeaning) arguments as model capabilities get better. I'm afraid you're asking for too much.
> admit the clearly intelligent machine is intelligent
citation needed
There is nothing indicating these models are clearly intelligent. Language fluency is not cognitive intelligence, and to think otherwise is falling into the trap of anthropomorphizing the LLMs.
They are still probabilistic engines, there is no causal reasoning still, they only emulate logic, and as far as we know, there is no agency, just the illusion of agency.
The danger here is not existential as you say. We aren't on the cusp of some machine uprising by super intelligence. The threats are algorithmic bias, misinformation at scale, and displacement of human labor.
lol sure buddy, keep telling yourself that
"This War Will Destabilize The Entire Mideast Region And Set Off A Global Shockwave Of Anti-Americanism vs. No It Won’t"
https://theonion.com/this-war-will-destabilize-the-entire-mi...
It's denialism, same as climate change, the subconscious fear to really grapple with the actual "what if" alternative scenario. Anthropic are true believers. They got to $1T in 5 years by being exceptionally smart and ahead of the curve here. Meanwhile HN just continually devolves into reactionary cynicism. "must be marketing, they just want to be rich, impossible AI advances much further." Meanwhile at every step of the way, Anthropic and X-riskers / "doomers" are vindicated in their correct predictive beliefs. We're headed to a future far dangerous than nukes very soon. We're in an arms race to detonate one 100x the size
I don’t really think it has to do with general denial in this case. I think he’s likely right in that pushback against datacenters is partially the release of angst over the threats which these models pose but in an indirect fashion since power is so unevenly distributed that local political organizing is the only real mechanism for people to act through. I think everyone feels nervous about these tools. People are scared of autonomous weapons, hackers using agents, and any of the other present issues that already exist. I think some of the pushback that comes in these threads is the fact that people like Dario are so focused on the long-term view that they suppress all immediate threats which already exist. You don’t need to argue about exponential growth or “AI being better than humans at everything” to want to better regulate this technology.
I would argue the best way to safeguard against long-term threats is to start by focusing on the issues that already exist. If you can offset the health risks of local datacenters or issues of unequal distribution of wealth by creating a more equal society right now then you’re already on the path to handling these long term issues. To me, this distant focus only distracts from the already present issues and conceals effective policy in this moment. We do need to safeguard against AI risk and it’s already here. Don’t even get me started on the havoc which recommendation systems have caused in society in the last 15 years which we still don’t call AI because it doesn’t speak.
Tl;Dr: These essays can feel disingenuous because:
1) AI risk is already present without exponential growth. The exponential growth argument often feels like a distraction from the fixes we could put in to fix the current issues that are already here.
2) The people stating this argument often have billions of dollars to gain if it comes true. While they may be altruistic, I also don’t see them doing all that much to fix the issues that people are already claiming exist and instead continue onwards on their path by justifying that they are the only ones responsible enough to handle it if the super intelligence does arrive. By continuing down that path, if that day ever does arrive they’ll have ensured the existence of a system which is unable to handle it.
I think we should treat this with smirking suspicions until their IPO happens.
I appreciate the critical perspective on political and economic power, as long as it's consequently followed, and every willingness for cooperation and the creation of fair rules is good.
What makes me doubt that Dario Amodei has really internalized the problem is the lack of humility, the stance that it's just important that the "good guys" keep the technology away from the "bad guys".
If you really want to provide AI with public benefit, you need to prevent power concentration. How? Some unpolished ideas, I'd be happy to hear yours:
- Avoid getting too close to an administration that is openly attacking democracy and is not interested in the benefit of humanity or mutually beneficial cooperation.
- Don't support surveillance. Non-(US-)Americans have human rights and privacy, too. Prepare for a situation where a government tries to convert your compute infrastructure into surveillance infrastructure.
- Support the creation of community data centers. In other words, build data centers together with local communities and make sure they profit from them.
- Advocate for laws that require transparency about resource usage and emissions of data centers.
- If you don't want an AI race, make sure that other countries don't need to fear the US concentrating too much power. Create institutions that can be trusted by other countries, too.
EDIT: I forgot:
- If qualified labor will actually turn out to get devalued, we also need a plan to prevent states from turning into rentier states that don't depend on a well-educated society any longer.
(On the other hand, I have been fooled by too many billionaires claiming to act in the interest of democracy and freedom. I once fell for a billionaire buying a social media platform, claiming to be a free speech fundamentalist, and it didn't age too well.)
Can we not open up every article talking to working professionals as if they’re children?
I like to stay up to date on things but more and more I’m finding myself pointing codex at a URL and saying “get to the point”.
He wrote it with ai that’s why
Sit back and watch the stock market implode as these lot double down on safety theatre. The hype is coming to an end.
They have to keep it up at least until the IPO dust settles.
> The hype is coming to an end
I'm sure you have evidence for this
My god this guy is insufferable. Stop mis-using the term exponential
Looks pretty exponential to me [1]. From a fully independent, non-profit research group.
[1] https://metr.org/time-horizons/
Release date seems like a terrible x axis with how much more compute they are using. Not to mention while I like what METR is trying to measure, it is an uber specific metric. And frankly, me just complaining, they’re prompts I feel do most of the work for the AI. I’ve never gotten as detailed instructions as they give the AI for the task
Whilst true, if you had unlimited compute 5 years ago, we wouldn't be anywhere near Mythos level purely because the technology behind the models wasn't refined enough.
Suggest a better name for what’s happening with LLMs please.
Snake oil scam.
They simply don't do what the label on the box says they do.
It is really hard to believe you actually believe this unless there really is this class of people that are so addicted to social media that they have confused performativity with actual thinking.
Anthropic reminds me of the Democratic Party in the days of Biden. Who is really running the show ?
Does it really help to say it's exponential?
People are not in charge, and have not been for some time.
Corporations and governments select their leaders and policies to advance their interests. People fit the work, not vice-versa. Only external competition or internal capabilities limit them (i.e., predation or resources).
External resources have been optimized as profit and exported costs; now that AI replaces the pesky need to source elites, capabilities will expand, which will result in more competition.
Law is not a lot more than settled expectations, and increasing capabilities changes expectations much more than even market disruption and disinformation, so I wouldn't expect law to save us.
As far as I can see, only competition will temper things, and only if AI companies are seen as responsible for their customers' applications - which I doubt. I say this in hope of being proven wrong.
The comments here. They make me feel that we are so doomed.
We all want to nuclear codes so badly. We are addicted to intelligence and labour so badly that we simply can't concieve that a pro-social actor might want us all not to have it, and for good reason.
I mean... Obviously, insiders like Oppenheimer (who dedicated their lives to considering the implications of the technology under discussion), they just feared nuclear proliferation because they wanted all the profits for themselves, right :(
Look - it's WAY more fun to just call Dario a goober than engage with the actual substance of the essay. Duh!
Sure, this may be the most important invention ever with near certainty to reshape society over the next few years, but meh. We should probably just immediately dismiss the concerns of anyone working on it without addressing their arguments at all. It's easy, we can justify ignoring their warnings by saying they're self-interested or too self-important or whatever.
Life is more fun when you live it with your eyes closed! You should try it out too.
Omg I love you. Thanks
"Democracies should seek to form a global coalition centered on building AI according to their common values, iteratively trying to draw in the rest of the world by making it more and more attractive to be part of the coalition and less and less attractive to be outside it."
It's not clear to me on which side of the coalition USA is meant to be in this divide. And as an European I'm not sure whether being in China's or USA's coalition is better in the long term.
In general, this deliberate mongering of ever more geopolitical division is extremely harmful. As is the Trump bootlicking.
[flagged]
[flagged]
Please don't post unsubstantive comments to Hacker News, regardless of who is a goober or you believe they are.
[flagged]
LOTR is easy to love. It's a classic story arc, well told, and well adapted for mass consumption.
It's also a very thinly veiled metaphor for the ceaseless pursuit of industrialization in every aspect of life chipping away at the humanity and metaphorical magic of the human experience, and the perils of embarking on that endeavour to the exclusion of everything else.
Personally, I find some humour in this situation.
Maybe Dario should have cited The Lorax.
AI is a lethal risk to mankind and should be totally rolled back for a century of close research before we try again.
What about punishment on AI exponential for security failures, electric chairs for CEO of high ranked frontier models that gets jailbroken by geopolitical adversaries.
> Ban the domestic use of fully autonomous weapons.
"Domestic" is an interesting shift here from the earlier _general_ discussions on autonomous weapons. So robot imperialism is good for Dario. But the imperial boomerang is a thing, in which case the proposed regulation itself is smoke and mirrors or just regulatory capture?