It's coming for everyone: there's a startup that introduced robot arms that promise to learn anything 1 shot: if it holds - and there's little wiggle room for one shot - it would be a good indicator that manual labor is also soon gone.
The only question that remains is if 99.9% of the world population will let themselves be fooled by UBI & VR to keep the 'masses content; while the 0.1% hold all the wealth and AI power.
And this is decided now, because it is too late once AI is consolidated under one bi/tri-llionaire.
I feel you skipped the step, that explains how this works in democracies. Musk was just not able to buy an election, and decidedly so.
It might all end very badly, but we should not be so prone to presume bad outcomes without explanations, that are just as good, as we would expect for good outcomes.
> Musk was just not able to buy an election, and decidedly so
This time he wasn't able to. But the time before (2024) he seemed to have some effect. I'm sure that he and the other oligarchs are going to learn from this midterm election and try different approaches in the future. Think of the 2026 election as one of his crashed rockets. They learn from those crashes - iterative improvement. They'll figure out ways to be more sneaky next time.
This exchange reads to me like a conversation from 1938 in Berlin. How dangerous are they, those sneaky Jews?
If you had millions of followers and billions of dollars, and you were convinced that America will go down if, say, Trump wins an election, would you not use the influence you have?
One could argue that you’d be a coward if you didn’t - having the conviction, having the means, but not wanting to stick out your neck.
Humans amassing money in an AI world is like squirrels amassing nuts when humans are logging the forest.
If we want to have a planet left we need to make it so laughably inefficient to run AI ground-side that no one bothers.
Extremely high temperature semiconductor research to run datacenters in dyson swarms around stars/black holes is fundamental research for human survival at this point.
Not a chance: if there was even a 1% chance of this being true you'd see something entirely different happening. If AI research self-reinforces, as Anthropic publishes (but doesn't actually believe, it can't), then it will win from all the economic activity on the planet. The major consequence of that is that money (as a source of labor, ie. to get people to do stuff for you, to get stuff for you) is now worthless.
Why? It is now easy to get a few motors to do that stuff for you. It is now easy to get whatever manufactured ... except for raw materials, and energy if you see that as separate.
Think about that again: if superintelligence exists the huge change is that money is now worthless and raw materials have all just turned more valuable than their weight in pure gold (assuming they're more useful than gold, which is all of them). Steel, just raw ingots, will increase in price BY A FACTOR ONE HUNDRED THOUSAND, minimum. At this point your dog can probably reason towards the investment opportunity, no?
You would buy farms, mines, mining rights, water, energy (and not for DCs, actual energy sources), rivers, raw materials ... that sort of thing. A superintelligence is not going to be exempt from the means of production. Raw materials are mostly monopolized in the current economic system, so this would in fact be absolutely critical to your success. And hoards of raw materials, anything, will be worth more than Gold (gold being pretty useless except for looking pretty and having a somewhat favorable electrical resistance, but not super great. When it comes to strength it is beaten by 2-day old chewing gum. So if economic activity is 100x-ing every raw material actually used for something will go into extreme shortage)
So no, sorry. Obviously this cannot be how Amodei, Altman, Google, Meta, and others are seeing things: they're going after money, not something else. So they are either stupid or they know 99%+ sure they do not have superintelligence. Because if they do let superintelligence loose, mining countries will have them by the balls and will squeeze. Hard. Yet the money all these people are collecting boatloads of will be useless. And yet, they want the money, not the raw materials! That tells you what they believe: that the money is more valuable than raw materials ... AND more valuable than superintelligence.
In other words: AI is a bubble, a hype, according to the management of the big AI firms. It's a way to get people to give them money, not an actual path to superintelligence. Either that, or these people are complete idiots.
Not really. You are assuming that ASI will just win irrespective of anything else, 2 seconds after it starts. How would it do that? By launching the nukes? And if it takes time, everyone else has time to catch up ... and raw materials will be the determining factor.
So that's not what happen. Yes, someone will be first, but the second will not be far behind.
Once you have ASI, raw material owners and the governments they're in will PICK the winner from the choices available. Just like they did in the 20th century.
To say nothing of the consequences of what is currently happening with Chinese models only 2 months behind frontier labs. Even if it's 8 months or 2 years or whatever. ASI will not build an insurmountable advantage in the world economy in 2 months, especially not if some raw materials players sabotage it (which seems likely in the extreme, and is simply what raw materials players are already doing, so they don't even have to realize what's happening and start doing it)
You are assuming that ASI won't just win, simply because you can't imagine it? Examples:
- Sophisticated cyberattacks to disable or quietly sabotage competing labs
- Undermining nuclear second-strike capability by locating submarines and infiltrating command-and-control networks
- Fast-replicating nanotechnology
- Superpersuasion (I am dubious on this)
etc., backed by global surveillance to preempt sabotage efforts. If you haven't read Superintelligence you should, all the leaders of the labs certainly have.
I just totally exclude that, because that would be a totally uncontrollable AI, which would be "huge meteor strikes the earth" type of scenario. Extinction level event stuff. If that happens, we're all dead, including Altman and Amodei, Google and Meta management.
So this level superintelligence also falls under the statement "if management believed that, this isn't what you'd see them do".
Plus your statement doesn't explain why they're after money.
You cannot control or align something 10x your intelligence. That would be like a dog controlling a human. For one thing, it won't even work after a short while because the dog's requests are going to be stupid. Except Altman and Amodei will be the dogs.
I'm with the people who really doubt alignment matters.
Besides, one thing keeps coming up in the open models: removing alignment makes them better, more interesting and more creative (difference for fiction, games, and ... well look at civitai is enormous).
And alignment also opens up an avenue to beat these models. And this will be a new level of moral dilemma: you can't judge, because the AI is too complex, too fast, too smart, ... and you cannot even react in time.
> ... they are after money because it allows them to scale and hopefully control AGI/ASI ...
Makes no sense for companies that are already making their own chips. If they truly believed they could figure it out FAST, they'd be buying sand (and it's not everywhere: apparently the purity is a big deal and not available in many places)
Is the equivalent of ants trying to outsmart Google that wants to build a datacenter on their land. ASI will unfathomably outsmart the wealthy and intelligent humans like Stockfish 13 does Magnus Carlsen
This is jumping the real world concerns into the realm of science fiction. How about we engage with the premise that before the AGI god comes into being and annoints the Rationalists as it's chosen people, there will be people who will try and use this to enrich themselves and immiserate others.
Every year we are making the "science fiction" of the previous year come true. The trajectory has been clear, the insider info from lab employees themselves are that the scaling laws continue unabated. The coding of the last 40 years is gone in a flash. Unanswered math problems are being solved every week and the top mathematicians of the world see crisis. The only "science fiction" is hyperbole about anointing rationalists, it's a strawman no one holds, X-risk concern is about disregarding everyone.
"there will be people who will try and use this to enrich themselves and immiserate others."
This is already happening from Big Tech and VC's, all the big money. You should try listening to the people that have shirked enormous fortunes in order to tell the truth about things. Some examples:
Sarcastically quipping that the AGI god would anoint the Rationalists as it's chosen people is like saying that the global warming god is going to come along and anoint the Climatologists as its chosen people. It's like saying Al Gore is a reptile and is excited for the universally hot, humid future.
Everybody who thinks they're "the chosen people" predicts the end of the world is coming, but not everybody who predicts the end of the world is coming thinks they're "the chosen people". Maybe the end of the world is coming and it's Just Bad.
I work as a research engineer within an Alphabet company, and I have been in the industry for almost 15 years. I am in the same place as Dwarkesh. I have seen some things that have absolutely shocked me within the last two months. A team of agents rewriting a model from scratch that had had many years worth of R&D spent on it, beating the original in just a few days. An agent that anyone in the company can just message an experiment id and say 'improve this', and it will come back a few hours later with like 20 issues, having examined checkpoint weights and gradient steps.
These agents know every paper, every trick, have amazing instincts, and can code up 20 ablations and start them running before you finish your commute. I'm finding guiding them is becoming less and less useful. I have never felt my obsolescence rushing towards me as much as now.
What options does humanity have? All I can think of is
1. Pause AI development and allow humans to stay in control a bit longer while we work on making AI safer, allowing the field to advance bit by bit (and hoping this plan even works)
2. Allow AI to progress to superintelligence and hope that it is aligned and will treat everyone fairly
3. Stop AI development altogether and create a world government panopticon which ensures it never progresses towards human obselesence?
This seems to be technically possible for uranium enrichment, but I'm not sure it's technically possible for ML research. You can do it on any CPU just really slowly, right? Or you could hide small numbers of GPUs in any basement. I think AI-2040 has an estimate of how much time we could possibly buy ourselves this way (even ignoring all the political problems), and I don't think it's especially long.
We are somewhat lucky that your option 1 will just happen by accident. And by this I mean the fall from grace of Jensen Huang will put a bit of brakes on the AI movement. It will be temporary, for sure, but it will give us some time to adjust.
One person's scandal won't put a break on AI progress itself, at worst it is viewed as an arbitrage opportunity to exploit short term chaos such that leadership restructuring puts different people in charge.
Obviously there is the general gradient between full speed ahead <-> full stop, but what's more important is the communication and assurances between all relevant parties (US/Chinese governments, major AI labs). If everyone is forced to make a deal without being aware of the risks, then there is no point as they will imagine that these safeguards are overly stringent.
> These agents know every paper, every trick, have amazing instincts, and can code up 20 ablations and start them running before you finish your commute. I'm finding guiding them is becoming less and less useful. I have never felt my obsolescence rushing towards me as much as now.
For math/cs research I found that agents alone doesn't replace a harness that gives them a strict sense of reality to have a better "compass". It's not only Lean but a plethora of tools like Mathematica, SageMath, SAT/SMV solvers, and very niche ones.
I am not in an alphabet lab, feel free to hire me though, and using off the shelf models with a few tweaks to the harness I used the same models to build I beat all the published benchmarks by substantial margins.
We need people at this point to tell the AI to do better and to stop disastrous side quests that kill its productivity.
And yes, rsi is here already. And its small enough that it can fit on a local gpu.
The tsunami is coming for everything, I now think that AI is the first democratising technology since the musket. And the results will probably be the same.
I think the next step for developers is to combine existing technologies. For example, in the medical imaging space that are thousands of published algorithms that were implemented once (by the researcher who published the paper), but never used. AI can now read the paper and one-shot an implementation. I do wonder if there are breakthroughs to be had by combining such algorithms.
Almost certainly yes - I’ve tried this for some imaging problems - but what tends to happen is you find that the method is fragile and only works well on a narrow task, or you need some specific data that’s a pain to label, or the results are strongly overfit. Reproducibility crisis type stuff. Which makes sense when you think about how narrowly scoped many papers are, and the fallibility of the average grad student. Also survivorship bias is strong. Lots of papers are forgotten about because the methods didn’t really work for other things. Sometimes it doesn’t matter, maybe it’s fine to have a method that only works on some niche type of data that you have.
But models are really good at being told “here is a paper, check the implementation against the claims in the paper and munge my data to work with it”. On the plus side, many seminal papers have really simple implementations. Like the meat of many CVPR papers is a derivative repo + one function modified. Often they can fix things or backport improvements from more recent work. And they can definitely re-implement things that ought to work, but the authors never published.
If AI is a verifiable domain, how do you get AIs to construct benchmarks to apply to themselves?
It seems that the only benchmarks that could be automatically constructed are those for tasks that are themselves verifiable (theorem proving, coding, etc).
Non-verifiable tasks require humans to define what exactly a good job is.
The RSI hypothesis is dependent on an assumption that progress on verifiable tasks generalizes to non-verifiable tasks.
There are hundreds of verifiable tasks in AI development as simple benchmarks on the models themselves, such as
1. Performance engineering ability
2. Ability to reconstruct visuals in code alone
3. Ability to orchestrate cheaper models towards a given task
4. Pretty much any verifiable engineering task
The loop is pretty clear, have AI's work on pre and post-training, then benchmark their outputs against these tasks to determine how to steer AI progress. The non-verifiable stuff isn't on the critical path towards these accoutrements anyway.
What I find fascinating about AI predictions is the human nature reflected behind them. There seem to be two camps: the first includes the author of this essay, people who are beginning to believe that unknown capabilities yet to exist will soon become possible and are impossible to predict. This camp seems to be winning an increasing number of people due to previous unknowns being solved very rapidly. Therefore, belief has been increasing in this group that AGI and other big AI questions such as massive human job losses can't be far around the corner.
On the other side of the spectrum we see people who still want evidence and kick their AI predictions out to a more distant future by pointing at current limitations. This group seems to move the goalposts as needed to support their argument with each new advancement. As the author of the article pointed out, he has moved from this camp to the first one. I have also moved to the first one.
Anthropic would not be IPO’ing if the first case were coming anytime soon. They’d have no need. Their current behavior looks more like a company who wants to cash out while they can.
The behavior of these frontier labs is much more telling than anything else.
It's human nature and all to common to move goalposts when a belief is entrenched or ideologically motivated. It can be as simple as the belief is aligned with the interests of that person.
Future capabilities are impossible to predict, therefore the timeline must be accelerated to an attention grabbing year such as 2030. All the same camp
the author bet everything on AI explosion and it paid off for him. Sometimes it is easier to be a believer and hope you end up right than endlessly discuss.
I've come to the conclusion that the only thing LLMs are good for is writing code and ONLY things they will excel at are things that can be solved by writing code.
All that fuss about the Mathematics problems solved was just LLMs writing Lean code. And besides that, agents are not anything remote to human in terms of behavior and accountability. They are like silly toddlers with a lot of knowledge.
I think AI is not coming for all white collar jobs, it's only coming for jobs that can be solved with code.
One can argue that they will be able to do brain surgery by writing code that execute instructions in a robot arm scalpel. Only time will tell.
it is reasonable to ask if a person has done something relevant to their opinion - lots of people have opinions on things that they don't understand. is there a benefit in giving credence to unsubstantiated opinions?
He's making an argument and you're free to engage with it or not, if you don't read anything by anyone who doesn't meet your credentials quota then save your time and skip it.
It's not only about verifiable or not. It's also about how high the fruit hangs vs how good your ladder (search heuristic) is. NP stuff is readily verifiable for example.
Absolutely plausible that this math avalanche was 1 cm deep, and harder problems are not solvable without new math theories that have to be constructed with a lot of understanding needed and taste (human heuristic).
Why is there a limit from the chips? If AI is better than all the humans in every field, then it should also be able to design a better chip no problem. It should be able to figure out how to make new manufacturing processes etc to make chips smaller and faster and overall more capable.
I feel like this is why more distributed control or open ownership of AI models is such an important goal for regulators. As AI accelerates its own development, the private companies that are initially maybe only a few months ahead of open-source will widen the gap more and more.
It seems likely we end up with an even more extreme tech oligarchy than we already have when superintelligence arrives. And that's only if we manage to solve alignment and it doesn't do something more horrible.
Dwarkesh has been skeptical about AI claims for a long time. In his interview with Dario earlier this year, he pressed him hard. So this is a fairly recent change of opinion.
What he says still doesn't map to my daily practice. I use Astra and Fable extensively and they need an incredible amount of guidance and harnessing to produce fairly simple systems. Any decent code is produced with tons of static tooling, agent files, harnessing, and even then, there are myriad mistakes. My main point of skepticism is whether software engineering is verifiable at all, once you factor in architecture, code design, let alone the human element in usability, requirements, etc.
As far as I'm concerned Dwarkesh could be prey of AI psychosis... I don't think a proxy metric like the average daily spend of AI researchers going up proves anything.
The technology has undoubtedly advanced but for me it's still a coin toss on whether we're in an exponential or an s-function.
Taking a step back, it's basically getting very good in one arena, with who knows how much money is being spent refining things. But nobody appears to be working on the other aspects of intelligence that I (as a lifelong sf reader) would expect from an AI, let alone a superintelligent one. There is no permanence of anything except at the "build the next model" level and some notes it keeps and can scan quickly. Instead of an intelligence that focuses on you for a moment, you're getting a new copy every time you interact with it after a certain amount of time. There is no "game loop". It's just there to respond to you and be ordered about. It will never reply in a text chat hours later wondering how your latest meeting went. It doesn't have a concept of concepts. It regurgitates words and code in helpful ways, but there is no understanding there. It doesn't know what it means when it says things, it's all just a (very impressive!) act.
I've been building neural nets in C as a past-time hobby, and I get the feeling that I've been peeking under the curtain too much. The magic is going away. I know they have been getting creative with training, different types of neural nets, and so on. I don't know that I want these organizations working on building an actual sf-like AI, because that might actually human-ending disastrous, but this side road they've gone down isn't going to get there from where I'm standing.
I'm also starting to worry that all I do is regurgitate words and have no real understanding either :)
That also matches my observation, particularly in economics once you factor in a scarce resource. But it's not like I have a convincing metamodel to say whether that or the other distribution is more common. But neither do people who fervently believe in the exponential...
I understood that the author was using the actual term, before it was hijacked by a malignant narcissist as a perversion of the meaning of words and an indicator of fealty (which only applies to Federal employees in the Executive Branch, by the way).
So? Are we all supposed to stop using the term, despite it being around since like the late 90s? Sorry, I don't let politicians control my life like that.
> When is superdupermegaintelligence going to be announced?
Isn't it already here? Infallible God-King Trump and all his ultra-genius tech-bro buddies delivered superintellgence already, and due to exponential growth it should already exist by this time, no? By this time next week it will have solved all problems and made humans obsolete.
So if AI is shortly (Dwarkesh suggests maybe next year) going to be better than any expert at anything, I guess I'll just wait until it's there and ask it to make me rich.
Seems far simpler than asking it to make super-human intelligence since nobody appears to know how to do that, and tons of people already know how to successfully start companies and make money.
My backup plan is to still be here in a years time, not rich, reading about the latest forecasts about how the singularity is coming real soon now.
"The avalanche that hit math is coming for X" is already a tired meme. It's telling that the impressive success of applying LLMs to the entirely abstract domain of math is being used to make extrapolations about all kinds of domains that aren't amenable to 10k agents (intelligently) throwing shit at the wall.
A lot of AI research can be already automated. The experimental part of work can be tonally outsourced to agents (at least for the topics that I am interested in). I had some success with outsourcing theoretical work as well. These days I use quite advanced orchestration. Interestingly, the most difficult part still is writing good stories for research papers. There I can still beat any AI. And when it comes to research we have been already covered by an avalanche of AI slop. There was always slop in AI research but now it is the fully automated slop. A few years ago there was some human-made slop, now it is being factory produced. AI can do good research on its own, but good research is rare in general.
>majority of this acceleration today is likely attributable to Anthropic’s extensive use of AI to do R&D
Lots of companies think they are reaping similar rewards from ai coding and research but it has failed to lift bottomlines or create any new products. My last two employers are gungho on ai and ppl have been producing unfathomable amount of ai output. To a causal observer it would look like we have really accelerated but this has had no visible impact on any of the products that employers produce other than make them shittier.
Whats different about product development at athropic that ai is particular suited for not generating slop.
I’m not into math. Certainly not professional math. Judging by the headlines here I know that AI (or AI + shepherds) solved some math. Some weeks later I saw headlines that mathematicians were psychologically in shambles. Um, okay? Maybe this was a long-term development, that AI has gradually made mathematicians (and not just the AI-pilled ones) realize that it’s better than them? That’s the only thing that makes sense. Because it doesn’t make sense to have a breakthrough in one week and then a little later have a collective depressive episode. No human institution is upended just like that.
Nothwithstanding genuine one-shotting, mathematics is certainly a great propaganda angle for the AI Bros. Ivory tower image. Not obviously directly applicable to the real world. Theoriticians solving theoretical problems because the problems are there and tenure. That’s the perception, anyway; highly-educated whiteboard writers, not salt of the earth workers. Then you frame that crowd as jealously guarding their theorycrafting fiefdom. But it’s no match for the great AI. And then they go into group therapy next week.
Isn't Sam Altman accused of being a habitual liar by a lot of people, including the richest man on the planet, the arguably best (certainly longest-lasting) AI researcher, the most famous AI ethics researcher, and half of his own board?
Strange how this guy can be CEO of anything. All you hear management types ever blubber about is integrity ... Clearly there's a problem here.
TLDR; someone used some fake graphs and probabilities pulled out of thin air to justify an unwarranted conclusion based on a false analogy to a situation that is far from clear.
It's coming for everyone: there's a startup that introduced robot arms that promise to learn anything 1 shot: if it holds - and there's little wiggle room for one shot - it would be a good indicator that manual labor is also soon gone.
The only question that remains is if 99.9% of the world population will let themselves be fooled by UBI & VR to keep the 'masses content; while the 0.1% hold all the wealth and AI power.
And this is decided now, because it is too late once AI is consolidated under one bi/tri-llionaire.
I feel you skipped the step, that explains how this works in democracies. Musk was just not able to buy an election, and decidedly so.
It might all end very badly, but we should not be so prone to presume bad outcomes without explanations, that are just as good, as we would expect for good outcomes.
> Musk was just not able to buy an election, and decidedly so
This time he wasn't able to. But the time before (2024) he seemed to have some effect. I'm sure that he and the other oligarchs are going to learn from this midterm election and try different approaches in the future. Think of the 2026 election as one of his crashed rockets. They learn from those crashes - iterative improvement. They'll figure out ways to be more sneaky next time.
This exchange reads to me like a conversation from 1938 in Berlin. How dangerous are they, those sneaky Jews?
If you had millions of followers and billions of dollars, and you were convinced that America will go down if, say, Trump wins an election, would you not use the influence you have?
One could argue that you’d be a coward if you didn’t - having the conviction, having the means, but not wanting to stick out your neck.
You might not be that different from “them”.
UBI is a political impossibility and VR still sucks. The old ways are not going anywhere.
The “old ways”? France 1790ish? Or do you mean the really old ways; sacrifice.
Don’t forget genocidal massacres and global omnidirectional holocausts as the have-nots compete for the right to exist.
AI doesn't need the 0.1% either.
Humans amassing money in an AI world is like squirrels amassing nuts when humans are logging the forest.
If we want to have a planet left we need to make it so laughably inefficient to run AI ground-side that no one bothers.
Extremely high temperature semiconductor research to run datacenters in dyson swarms around stars/black holes is fundamental research for human survival at this point.
Not a chance: if there was even a 1% chance of this being true you'd see something entirely different happening. If AI research self-reinforces, as Anthropic publishes (but doesn't actually believe, it can't), then it will win from all the economic activity on the planet. The major consequence of that is that money (as a source of labor, ie. to get people to do stuff for you, to get stuff for you) is now worthless.
Why? It is now easy to get a few motors to do that stuff for you. It is now easy to get whatever manufactured ... except for raw materials, and energy if you see that as separate.
Think about that again: if superintelligence exists the huge change is that money is now worthless and raw materials have all just turned more valuable than their weight in pure gold (assuming they're more useful than gold, which is all of them). Steel, just raw ingots, will increase in price BY A FACTOR ONE HUNDRED THOUSAND, minimum. At this point your dog can probably reason towards the investment opportunity, no?
You would buy farms, mines, mining rights, water, energy (and not for DCs, actual energy sources), rivers, raw materials ... that sort of thing. A superintelligence is not going to be exempt from the means of production. Raw materials are mostly monopolized in the current economic system, so this would in fact be absolutely critical to your success. And hoards of raw materials, anything, will be worth more than Gold (gold being pretty useless except for looking pretty and having a somewhat favorable electrical resistance, but not super great. When it comes to strength it is beaten by 2-day old chewing gum. So if economic activity is 100x-ing every raw material actually used for something will go into extreme shortage)
So no, sorry. Obviously this cannot be how Amodei, Altman, Google, Meta, and others are seeing things: they're going after money, not something else. So they are either stupid or they know 99%+ sure they do not have superintelligence. Because if they do let superintelligence loose, mining countries will have them by the balls and will squeeze. Hard. Yet the money all these people are collecting boatloads of will be useless. And yet, they want the money, not the raw materials! That tells you what they believe: that the money is more valuable than raw materials ... AND more valuable than superintelligence.
In other words: AI is a bubble, a hype, according to the management of the big AI firms. It's a way to get people to give them money, not an actual path to superintelligence. Either that, or these people are complete idiots.
If you believe in ASI but you spend all your money on farmland, mining rights, and power plants, instead of compute, you will lose the race to ASI.
Your argument only makes sense in a world with one AI lab. Labs are pouring all of their money into scaling because there is no alternative.
Not really. You are assuming that ASI will just win irrespective of anything else, 2 seconds after it starts. How would it do that? By launching the nukes? And if it takes time, everyone else has time to catch up ... and raw materials will be the determining factor.
So that's not what happen. Yes, someone will be first, but the second will not be far behind.
Once you have ASI, raw material owners and the governments they're in will PICK the winner from the choices available. Just like they did in the 20th century.
To say nothing of the consequences of what is currently happening with Chinese models only 2 months behind frontier labs. Even if it's 8 months or 2 years or whatever. ASI will not build an insurmountable advantage in the world economy in 2 months, especially not if some raw materials players sabotage it (which seems likely in the extreme, and is simply what raw materials players are already doing, so they don't even have to realize what's happening and start doing it)
You are assuming that ASI won't just win, simply because you can't imagine it? Examples:
- Sophisticated cyberattacks to disable or quietly sabotage competing labs
- Undermining nuclear second-strike capability by locating submarines and infiltrating command-and-control networks
- Fast-replicating nanotechnology
- Superpersuasion (I am dubious on this)
etc., backed by global surveillance to preempt sabotage efforts. If you haven't read Superintelligence you should, all the leaders of the labs certainly have.
I just totally exclude that, because that would be a totally uncontrollable AI, which would be "huge meteor strikes the earth" type of scenario. Extinction level event stuff. If that happens, we're all dead, including Altman and Amodei, Google and Meta management.
So this level superintelligence also falls under the statement "if management believed that, this isn't what you'd see them do".
Plus your statement doesn't explain why they're after money.
That doesn't follow. If the AI is 'aligned' to its creators, it will be able to disable competitors without causing mass extinction.
Consider that they are after money because it allows them to scale and hopefully control AGI/ASI
You cannot control or align something 10x your intelligence. That would be like a dog controlling a human. For one thing, it won't even work after a short while because the dog's requests are going to be stupid. Except Altman and Amodei will be the dogs.
I'm with the people who really doubt alignment matters.
Besides, one thing keeps coming up in the open models: removing alignment makes them better, more interesting and more creative (difference for fiction, games, and ... well look at civitai is enormous).
And alignment also opens up an avenue to beat these models. And this will be a new level of moral dilemma: you can't judge, because the AI is too complex, too fast, too smart, ... and you cannot even react in time.
> ... they are after money because it allows them to scale and hopefully control AGI/ASI ...
Makes no sense for companies that are already making their own chips. If they truly believed they could figure it out FAST, they'd be buying sand (and it's not everywhere: apparently the purity is a big deal and not available in many places)
>You cannot control or align something 10x your intelligence.
That's what you (and I mostly) believe. AI researchers do not believe this.
There's one guy playing that game!
https://www.forbes.com/sites/arielshapiro/2021/01/14/america...
This comment makese sense if we do extreme amount of reality suspension "think what will be possible in future..."
But we've all seen robots fold clothes to do that sort of thinking easily. Oh maybe we need to hangout on ai twitter more.
> the 0.1% hold all the wealth and AI power
Is the equivalent of ants trying to outsmart Google that wants to build a datacenter on their land. ASI will unfathomably outsmart the wealthy and intelligent humans like Stockfish 13 does Magnus Carlsen
This is jumping the real world concerns into the realm of science fiction. How about we engage with the premise that before the AGI god comes into being and annoints the Rationalists as it's chosen people, there will be people who will try and use this to enrich themselves and immiserate others.
Every year we are making the "science fiction" of the previous year come true. The trajectory has been clear, the insider info from lab employees themselves are that the scaling laws continue unabated. The coding of the last 40 years is gone in a flash. Unanswered math problems are being solved every week and the top mathematicians of the world see crisis. The only "science fiction" is hyperbole about anointing rationalists, it's a strawman no one holds, X-risk concern is about disregarding everyone.
"there will be people who will try and use this to enrich themselves and immiserate others."
This is already happening from Big Tech and VC's, all the big money. You should try listening to the people that have shirked enormous fortunes in order to tell the truth about things. Some examples:
https://x.com/hilbertspaess/status/2097476196791709843?lang=...
https://fortune.com/2026/09/26/geoffrey-hinton-godfather-of-...
https://yoshuabengio.org/en/blog/faq-catastrophic-ai-risks
https://ai-2027.com/
Sarcastically quipping that the AGI god would anoint the Rationalists as it's chosen people is like saying that the global warming god is going to come along and anoint the Climatologists as its chosen people. It's like saying Al Gore is a reptile and is excited for the universally hot, humid future.
Everybody who thinks they're "the chosen people" predicts the end of the world is coming, but not everybody who predicts the end of the world is coming thinks they're "the chosen people". Maybe the end of the world is coming and it's Just Bad.
I work as a research engineer within an Alphabet company, and I have been in the industry for almost 15 years. I am in the same place as Dwarkesh. I have seen some things that have absolutely shocked me within the last two months. A team of agents rewriting a model from scratch that had had many years worth of R&D spent on it, beating the original in just a few days. An agent that anyone in the company can just message an experiment id and say 'improve this', and it will come back a few hours later with like 20 issues, having examined checkpoint weights and gradient steps.
These agents know every paper, every trick, have amazing instincts, and can code up 20 ablations and start them running before you finish your commute. I'm finding guiding them is becoming less and less useful. I have never felt my obsolescence rushing towards me as much as now.
What options does humanity have? All I can think of is
1. Pause AI development and allow humans to stay in control a bit longer while we work on making AI safer, allowing the field to advance bit by bit (and hoping this plan even works)
2. Allow AI to progress to superintelligence and hope that it is aligned and will treat everyone fairly
3. Stop AI development altogether and create a world government panopticon which ensures it never progresses towards human obselesence?
> create a world government panopticon
This seems to be technically possible for uranium enrichment, but I'm not sure it's technically possible for ML research. You can do it on any CPU just really slowly, right? Or you could hide small numbers of GPUs in any basement. I think AI-2040 has an estimate of how much time we could possibly buy ourselves this way (even ignoring all the political problems), and I don't think it's especially long.
We are somewhat lucky that your option 1 will just happen by accident. And by this I mean the fall from grace of Jensen Huang will put a bit of brakes on the AI movement. It will be temporary, for sure, but it will give us some time to adjust.
One person's scandal won't put a break on AI progress itself, at worst it is viewed as an arbitrage opportunity to exploit short term chaos such that leadership restructuring puts different people in charge.
Plan A: https://ai-2040.com/
You reckon there are maybe a few things in between those three points?
Obviously there is the general gradient between full speed ahead <-> full stop, but what's more important is the communication and assurances between all relevant parties (US/Chinese governments, major AI labs). If everyone is forced to make a deal without being aware of the risks, then there is no point as they will imagine that these safeguards are overly stringent.
> These agents know every paper, every trick, have amazing instincts, and can code up 20 ablations and start them running before you finish your commute. I'm finding guiding them is becoming less and less useful. I have never felt my obsolescence rushing towards me as much as now.
For math/cs research I found that agents alone doesn't replace a harness that gives them a strict sense of reality to have a better "compass". It's not only Lean but a plethora of tools like Mathematica, SageMath, SAT/SMV solvers, and very niche ones.
I am not a research engineer and have only a vague idea what you are talking about. I doubt me plus a model can replace you.
Tell me more about how AI is able to fix security bugs within code.
I am not in an alphabet lab, feel free to hire me though, and using off the shelf models with a few tweaks to the harness I used the same models to build I beat all the published benchmarks by substantial margins.
We need people at this point to tell the AI to do better and to stop disastrous side quests that kill its productivity.
And yes, rsi is here already. And its small enough that it can fit on a local gpu.
The tsunami is coming for everything, I now think that AI is the first democratising technology since the musket. And the results will probably be the same.
I think the next step for developers is to combine existing technologies. For example, in the medical imaging space that are thousands of published algorithms that were implemented once (by the researcher who published the paper), but never used. AI can now read the paper and one-shot an implementation. I do wonder if there are breakthroughs to be had by combining such algorithms.
Almost certainly yes - I’ve tried this for some imaging problems - but what tends to happen is you find that the method is fragile and only works well on a narrow task, or you need some specific data that’s a pain to label, or the results are strongly overfit. Reproducibility crisis type stuff. Which makes sense when you think about how narrowly scoped many papers are, and the fallibility of the average grad student. Also survivorship bias is strong. Lots of papers are forgotten about because the methods didn’t really work for other things. Sometimes it doesn’t matter, maybe it’s fine to have a method that only works on some niche type of data that you have.
But models are really good at being told “here is a paper, check the implementation against the claims in the paper and munge my data to work with it”. On the plus side, many seminal papers have really simple implementations. Like the meat of many CVPR papers is a derivative repo + one function modified. Often they can fix things or backport improvements from more recent work. And they can definitely re-implement things that ought to work, but the authors never published.
If AI is a verifiable domain, how do you get AIs to construct benchmarks to apply to themselves?
It seems that the only benchmarks that could be automatically constructed are those for tasks that are themselves verifiable (theorem proving, coding, etc).
Non-verifiable tasks require humans to define what exactly a good job is.
The RSI hypothesis is dependent on an assumption that progress on verifiable tasks generalizes to non-verifiable tasks.
There are hundreds of verifiable tasks in AI development as simple benchmarks on the models themselves, such as
1. Performance engineering ability 2. Ability to reconstruct visuals in code alone 3. Ability to orchestrate cheaper models towards a given task 4. Pretty much any verifiable engineering task
The loop is pretty clear, have AI's work on pre and post-training, then benchmark their outputs against these tasks to determine how to steer AI progress. The non-verifiable stuff isn't on the critical path towards these accoutrements anyway.
I think more accurately, it rests on (lots of) non-verifiable tasks sitting in latent space between verifiable tasks.
It’s a small distinction, but I think it creates a meaningfully less challenging requirement.
What I find fascinating about AI predictions is the human nature reflected behind them. There seem to be two camps: the first includes the author of this essay, people who are beginning to believe that unknown capabilities yet to exist will soon become possible and are impossible to predict. This camp seems to be winning an increasing number of people due to previous unknowns being solved very rapidly. Therefore, belief has been increasing in this group that AGI and other big AI questions such as massive human job losses can't be far around the corner.
On the other side of the spectrum we see people who still want evidence and kick their AI predictions out to a more distant future by pointing at current limitations. This group seems to move the goalposts as needed to support their argument with each new advancement. As the author of the article pointed out, he has moved from this camp to the first one. I have also moved to the first one.
Anthropic would not be IPO’ing if the first case were coming anytime soon. They’d have no need. Their current behavior looks more like a company who wants to cash out while they can.
The behavior of these frontier labs is much more telling than anything else.
It's human nature and all to common to move goalposts when a belief is entrenched or ideologically motivated. It can be as simple as the belief is aligned with the interests of that person.
String theory is one that comes to mind.
Three camps. "Impossible to predict" one doesn't throw around numbers like 2030.
Future capabilities are impossible to predict, therefore the timeline must be accelerated to an attention grabbing year such as 2030. All the same camp
the author bet everything on AI explosion and it paid off for him. Sometimes it is easier to be a believer and hope you end up right than endlessly discuss.
I've come to the conclusion that the only thing LLMs are good for is writing code and ONLY things they will excel at are things that can be solved by writing code.
All that fuss about the Mathematics problems solved was just LLMs writing Lean code. And besides that, agents are not anything remote to human in terms of behavior and accountability. They are like silly toddlers with a lot of knowledge.
I think AI is not coming for all white collar jobs, it's only coming for jobs that can be solved with code.
One can argue that they will be able to do brain surgery by writing code that execute instructions in a robot arm scalpel. Only time will tell.
Sorry for the rambling, just puking my thoughts.
What has this author done that would suggest we should listen to his opinion?
It's the internet, you're not forced to read anyone's opinion.
If someone has interesting ideas, or a reasoned argument, do you require their credentials before considering what they've written?
it is reasonable to ask if a person has done something relevant to their opinion - lots of people have opinions on things that they don't understand. is there a benefit in giving credence to unsubstantiated opinions?
Most people here will know who Dwarkesh is, he hosts a tech podcast
That's right, most people know that. Why does popularity make his opinion valid? Citation: the US president.
What has he done other than host a podcast?
What have you done?
Not much but I've been a S. Eng for 10 years and I'm wondering where Dwarkesh's claim to expertise comes from.
> I'm wondering where Dwarkesh's claim to expertise comes from.
Seemingly from you, as a strawman. He doesn't seem to make such a claim in the article.
Not from me, although I suppose I did assume.
So he's not an expert. Just a guy off the street. Got it, I'll pass on the podcast then thanks.
And absolutely nothing was lost.
“Great minds discuss ideas; average minds discuss events; small minds discuss people.” -- Eleanor Roosevelt
That could only be true if there was nothing to gain in the first place
Just keep telling yourself that. ;)
[dead]
He's making an argument and you're free to engage with it or not, if you don't read anything by anyone who doesn't meet your credentials quota then save your time and skip it.
It's not only about verifiable or not. It's also about how high the fruit hangs vs how good your ladder (search heuristic) is. NP stuff is readily verifiable for example.
Absolutely plausible that this math avalanche was 1 cm deep, and harder problems are not solvable without new math theories that have to be constructed with a lot of understanding needed and taste (human heuristic).
Why is there a limit from the chips? If AI is better than all the humans in every field, then it should also be able to design a better chip no problem. It should be able to figure out how to make new manufacturing processes etc to make chips smaller and faster and overall more capable.
I feel like this is why more distributed control or open ownership of AI models is such an important goal for regulators. As AI accelerates its own development, the private companies that are initially maybe only a few months ahead of open-source will widen the gap more and more.
It seems likely we end up with an even more extreme tech oligarchy than we already have when superintelligence arrives. And that's only if we manage to solve alignment and it doesn't do something more horrible.
The machine of AI influencers is running at overdrive. A true avalanche of bullshit.
Dwarkesh has been skeptical about AI claims for a long time. In his interview with Dario earlier this year, he pressed him hard. So this is a fairly recent change of opinion.
What he says still doesn't map to my daily practice. I use Astra and Fable extensively and they need an incredible amount of guidance and harnessing to produce fairly simple systems. Any decent code is produced with tons of static tooling, agent files, harnessing, and even then, there are myriad mistakes. My main point of skepticism is whether software engineering is verifiable at all, once you factor in architecture, code design, let alone the human element in usability, requirements, etc.
As far as I'm concerned Dwarkesh could be prey of AI psychosis... I don't think a proxy metric like the average daily spend of AI researchers going up proves anything.
The technology has undoubtedly advanced but for me it's still a coin toss on whether we're in an exponential or an s-function.
Taking a step back, it's basically getting very good in one arena, with who knows how much money is being spent refining things. But nobody appears to be working on the other aspects of intelligence that I (as a lifelong sf reader) would expect from an AI, let alone a superintelligent one. There is no permanence of anything except at the "build the next model" level and some notes it keeps and can scan quickly. Instead of an intelligence that focuses on you for a moment, you're getting a new copy every time you interact with it after a certain amount of time. There is no "game loop". It's just there to respond to you and be ordered about. It will never reply in a text chat hours later wondering how your latest meeting went. It doesn't have a concept of concepts. It regurgitates words and code in helpful ways, but there is no understanding there. It doesn't know what it means when it says things, it's all just a (very impressive!) act.
I've been building neural nets in C as a past-time hobby, and I get the feeling that I've been peeking under the curtain too much. The magic is going away. I know they have been getting creative with training, different types of neural nets, and so on. I don't know that I want these organizations working on building an actual sf-like AI, because that might actually human-ending disastrous, but this side road they've gone down isn't going to get there from where I'm standing.
I'm also starting to worry that all I do is regurgitate words and have no real understanding either :)
Thank you for saying this. Sometimes I'm like, wow, this thing is way smarter than me, and others I'm wondering how it can be so dumb.
In my experience, s-curves are much more prevalent in nature than exponentials.
Counterpoint: A human lifespan on a thousand-year S-curve would see only exponential.
That also matches my observation, particularly in economics once you factor in a scarce resource. But it's not like I have a convincing metamodel to say whether that or the other distribution is more common. But neither do people who fervently believe in the exponential...
Why is dwarkesh using the Trump term superintelligence. This is a bad look for him.
He's trying to predict when it could happen, not using it as a replacement for 'AI'.
We already have a term for that- AGI that everyone has been using.
ASI and AGI are separate terms.
It was a commonly used term for AGI in 2014 hence this book ttps://en.wikipedia.org/wiki/Superintelligence:_Paths,_Dangers,_Strategies
I understood that the author was using the actual term, before it was hijacked by a malignant narcissist as a perversion of the meaning of words and an indicator of fealty (which only applies to Federal employees in the Executive Branch, by the way).
Since when do we give Trump the credit for the term superintelligence? An absolutely bonkers comment to make.
The insane part is mandating the term instead of AI.
He literally mandated that everyone used the term Superintelligence instead of artificial intelligence like two weeks ago
https://www.politico.com/news/2026/09/29/trump-signs-executi...
Just so his friends can sell you those sweet .si domains they got 100x cheaper
So? Are we all supposed to stop using the term, despite it being around since like the late 90s? Sorry, I don't let politicians control my life like that.
No one was using it. Everyone was using AGI.
When is superdupermegaintelligence going to be announced?
Well, we already have superdupermegalomania.
Just before hypergigaintelligence
> When is superdupermegaintelligence going to be announced?
Isn't it already here? Infallible God-King Trump and all his ultra-genius tech-bro buddies delivered superintellgence already, and due to exponential growth it should already exist by this time, no? By this time next week it will have solved all problems and made humans obsolete.
So if AI is shortly (Dwarkesh suggests maybe next year) going to be better than any expert at anything, I guess I'll just wait until it's there and ask it to make me rich.
Seems far simpler than asking it to make super-human intelligence since nobody appears to know how to do that, and tons of people already know how to successfully start companies and make money.
My backup plan is to still be here in a years time, not rich, reading about the latest forecasts about how the singularity is coming real soon now.
"The avalanche that hit math is coming for X" is already a tired meme. It's telling that the impressive success of applying LLMs to the entirely abstract domain of math is being used to make extrapolations about all kinds of domains that aren't amenable to 10k agents (intelligently) throwing shit at the wall.
Who sponsors this advertisement? Many AI boosters.
https://sponsorradar.com/channels/dwarkeshpatel
And that does not include Substack bulk subscriptions.
A lot of AI research can be already automated. The experimental part of work can be tonally outsourced to agents (at least for the topics that I am interested in). I had some success with outsourcing theoretical work as well. These days I use quite advanced orchestration. Interestingly, the most difficult part still is writing good stories for research papers. There I can still beat any AI. And when it comes to research we have been already covered by an avalanche of AI slop. There was always slop in AI research but now it is the fully automated slop. A few years ago there was some human-made slop, now it is being factory produced. AI can do good research on its own, but good research is rare in general.
At last, he's talking his book.
Hype hype hooray
>majority of this acceleration today is likely attributable to Anthropic’s extensive use of AI to do R&D
Lots of companies think they are reaping similar rewards from ai coding and research but it has failed to lift bottomlines or create any new products. My last two employers are gungho on ai and ppl have been producing unfathomable amount of ai output. To a causal observer it would look like we have really accelerated but this has had no visible impact on any of the products that employers produce other than make them shittier.
Whats different about product development at athropic that ai is particular suited for not generating slop.
I’m not into math. Certainly not professional math. Judging by the headlines here I know that AI (or AI + shepherds) solved some math. Some weeks later I saw headlines that mathematicians were psychologically in shambles. Um, okay? Maybe this was a long-term development, that AI has gradually made mathematicians (and not just the AI-pilled ones) realize that it’s better than them? That’s the only thing that makes sense. Because it doesn’t make sense to have a breakthrough in one week and then a little later have a collective depressive episode. No human institution is upended just like that.
Nothwithstanding genuine one-shotting, mathematics is certainly a great propaganda angle for the AI Bros. Ivory tower image. Not obviously directly applicable to the real world. Theoriticians solving theoretical problems because the problems are there and tenure. That’s the perception, anyway; highly-educated whiteboard writers, not salt of the earth workers. Then you frame that crowd as jealously guarding their theorycrafting fiefdom. But it’s no match for the great AI. And then they go into group therapy next week.
love reading about idiots that swallowed trumps "superintelligence" drivel. Altman said we had AGI years ago, now we wait for the "superintelligence"
Superintelligence is not Trump's concept, it was invented years ago.
Trump's mistake is using the word to denote AI in general when it already had an established meaning.
Isn't Sam Altman accused of being a habitual liar by a lot of people, including the richest man on the planet, the arguably best (certainly longest-lasting) AI researcher, the most famous AI ethics researcher, and half of his own board?
Strange how this guy can be CEO of anything. All you hear management types ever blubber about is integrity ... Clearly there's a problem here.
[flagged]
TLDR; someone used some fake graphs and probabilities pulled out of thin air to justify an unwarranted conclusion based on a false analogy to a situation that is far from clear.
Has Dwarkesh ever done a single rigorous analysis of anything?
Yes, of his viewer demographics:
https://www.dwarkesh.com/p/advertise
Yes.