This is more interesting and deserves better discussion than we got from the previous title, which was derailed by the "AGI" bit, so I replaced the title with a representative sentence from the video.
(Edit: plus a question mark, as we sometimes do with contentious titles.)
How different are world models from LLMs? I'm not in the AI space but follow it here. I always assumed they belonged to the same "family" of tech and were more similar than different.
But are they sufficiently different that stalling progress in one doesn't imply stalling progress in the other?
Depends if you’re asking about real world models or synthetic AI world models.
One of them only exists in species with a long evolutionary history of survivorship (and death) over generations living in the world being modeled.
There’s a sense of “what it’s like to be” a thing. That’s still a big question mark in my mind, whether AI will ever have any sense of what it’s like to be human, any more than humans know what it’s like to be a bat or a dolphin.
You know what it’s like for the cool breeze to blow across your face on a nice day. You could try explaining that to a dolphin, assuming we can communicate one day, but they won’t know what it’s like from any amount of words. That seems like something in the area of neuralink or similar.
The world models are not really useful yet. So they are starting lower, compared to LLM. So they probably have some decent gains to make still, before it gets really hard (diminishing returns).
On the one hand, that isn't necessarily a problem. It can be just a useful algorithm for tool calling or whatever.
On the other hand, if you're telling your investors that AGI is about two years away, then you can only do that for a few years. Rumor has it that such claims were made? Hopefully no big investors actually believed that.
The real question to be asking is, based on current applications of LLMs, can one pay for the hardware to sustain it? The comparison to smartphones is apt; by the time we got to the "Samsung Galaxy" phase, where only incremental improvements were coming, the industry was making a profit on each phone sold. Are any of the big LLMs actually profitable yet? And if they are, do they have any way to keep the DeepSeeks of the world from taking it away?
What happens if you built your business on a service that turns out to be hugely expensive to run and not profitable?
>On the other hand, if you're telling your investors that AGI is about two years away, then you can only do that for a few years.
Musk has been doing this with autonomous driving since 2015. Machine learning has enough hype surrounding it that you have to embellish to keep up with every other company's ridiculous claims.
I doubt this was the main driver for the investors. People were buying Tesla even without it.
Whether there is hype or not, the laws of money remain the same. If you invest and don’t get expected returns, you will be eventually concerned and will do something about it.
Lying to investors is illegal, and investors have incentive and means to sue if they think they were defrauded. The problem is proving it. I'm sure a lot of founders genuinely believe AGI is about to appear out of thin air, so they're technically not lying. Even the cynical ones who say whatever they think investors want to hear are hard to catch in a lie. It's not really about being rich and powerful. That's just the unfortunate reality of rhetoric.
In addition to the other comments/answers to this, I would like to add that if you lie to your investors (in public), and they suspect you're lying but also think it will allow you to cash out before the lie becomes apparent, they may not care, especially if the lie is difficult to distinguish from pathological levels of optimism.
Predictions about the future and puffery are not illegal. Lying about facts are. Nobody knows how far away AGI is, everyone just has their own predictions.
It's not a crime to be wrong; it's only a crime to deliberately lie. And unless there's an email saying "haha we're lying to our investors", it's just not easy to prove.
I mean there are different definitions on what to call an AGI. Most of the time people don't specify which one they use.
For me an AGI would mean truly at least human level as in "this clearly has a consciousness paired with knowledge", a.k.a. a person. In that case, what do the investors expect? Some sort of slave market of virtual people to exploit?
OpenAI defines AGI as a "highly autonomous system that outperforms humans at most economically valuable work" [0]. It may not be the most satisfying definition, but it is practical and a good goal to aim for if you are an AI company.
My personal definition is "The ability to form models from observations and extrapolate from them."
LLMs are great at forming models of language from observations of language and extrapolating language constructs from them. But to get general intelligence we're going to have to let an AI build their models from direct measurements of reality.
They really aren't even great at forming models of language. They are a single model of language. They don't build models, much less use those models. See, for example, ARC-AGI 1 and 2. They only performed ARC 1 decently [0] with additional training, and are failing miserably on ARC 2. That's not even getting to ARC 3.
> Note on "tuned": OpenAI shared they trained the o3 we tested on 75% of the Public Training set. They have not shared more details. We have not yet tested the ARC-untrained model to understand how much of the performance is due to ARC-AGI data.
... Clearly not able to reason about the problems without additional training. And no indication that the additional training didn't include some feature extraction, scaffolding, RLHF, etc created by human intelligence. Impressive that fine tuning can get >85%, but it's still additional human directed training and not self contained intelligence at the level of performance reported. The blog was very generous making the undefined "fine tuning" a footnote and praising the results as if they were directly from the model that would have cost > $65,000 to run.
Edit: to be clear, I understand LLMs are a huge leap forward in AI research and possibly the first models that can provide useful results across multiple domains without being retrained. But they're still not creating their own models, even of language.
Med-Gemini is clearly intelligent, but equally clearly it is an inhuman intelligence with different failure modes from human intelligence.
If we say Med-Gemini is not intelligent, we will end up having to concede that actually it is intelligent. And the danger of this concession is that we will under-estimate how different it is from human intelligence and then get caught out by inhuman failures.
I did initially encounter it on LessWrong and modified it slightly according to my preference. Did he coin the term? There are a lot of ideas (not inappropriately) presented without attribution in that context.
To share my experience, 25 years ago I looked into AI and had inclinations to see what scaling compute would do. It took no time to find advisors who told me the whole program I had in mind could not gain ethics approval and was mathematically limited. The former road block is now lifted due to the fact that nobody cares about ethics any more, the latter seems to be the remaining hurdle.
The goal of economic is not to reach AGI. It would solve the problems we have with the current market, therefore would it make less money, then to just "chase" for the AGI. Shirky principle in a nutshell.
It is critical to remember that there is a market for people who say "AGI is not coming"
It doesn't matter whether they are lying. People want to hear it. It's comforting. So the market fills the void, and people get views and money for saying it.
Don't use the fact that people are saying it, as evidence that it is true.
I guess the AI skeptic version of MIRI would be like, an organization that tries to anticipate possible future large problems that could arise from people anticipating an AGI that never arrives, but which people might believe has arrived, and proposes methods to attempt to prevent or mitigate those potential problems?
Oddly, in a bubble the highest valuations come just before the burst. This is obvious mathematical certainty that can be read by anyone viewing an exponential growth curve.
I mean that as valuations rise before a bubble bursts, the curve provides that successive values grow, sometimes at an increasing rate. The greatest values, and change in values comes just before the bubble bursts. The point being that high valuations and increasing valuations are not very capable of distinguishing bubble/non-bubble. In fact, tulips were most valuable, and those values were climbing at the highest rate before the tulip bubble burst.
I said that high valuation or valuations rising at an increasing rate is common to bubbles. This clearly refers back to the comment about the current high and rising valuations for AGI prospects.
I see that. I just have a hard time seeing how it is related to my question. Euclidean geometry was an "obvious mathematical certainty". Until it wasn't, and we learned that there are no "obvious mathematical certainties".
So I wanted to understand what you mean by this concept.
But we can leave it. It's not important. No harm meant.
There are mathematical certainties. In this instance we have that all exponentially growing bubbles will necessarily display the property that values grow and at an increasing rate just before the bubble bursts. This is so fundamental that I can't believe you don't see it. You can construct the growth curve out of this very fact alone. Therefore, the simple fact of growing values and increasing rates of growth in values is not determining bubble/non-bubble.
In other words, the statement "I know I am not chasing a bubble because valuations are high and growing at an increasing rate" is categorically false because this property is a common indicator of the exact opposite; that you are in an exponentially growing bubble.
Look, neither did I say there are no mathematical certainties, nor did I anywhere argue about the content of what you said.
The only thing I said is: there are probably no _obvious_ mathematical certainties in reply to what you said, as the history of mathematics shows with plenty of examples.
But it's already too much ink spilled for such a little difference.
I reckon now that you use words like "obvious" (or "categorically" in the last reply [1]) as markers of belief strength or something. That's fine, many people do it.
> "Look, neither did I say there are no mathematical certainties, nor did I anywhere argue about the content of what you said
The only thing I said is: there are probably no _obvious_ mathematical certainties in reply to what you said, as the history of mathematics shows with plenty of examples."
There is nothing more obvious and certain mathematically than the fact that exponential growth of bubbles entails increasing valuations and increasing rates of growth in those valuations.
I use the word "obvious" quite properly as in "can be read from any graph", or "is contained within the structure or the model". I use categorical because the implied conclusion of the original comment is a categorical error.
But I think I should be giving a reply also to the content of what you say. That's only fair after you took so much time explaining.
I think you conflate two questions in your statement: the question of what can be said about a function that is up to a point exponential and growing, and after that point decaying in an unspecified manner, but so that this point is a local maximum. And secondly the question if this is a good/the only/the best model for valuations in what is called an investment bubble that then bursts.
This conflation of model and modeled makes it hard for me to assess the argument, because I need to take these layers apart, and that involves guess work.
There are no layers and I am trying to help you in your difficulty. It is not sound to look around and observe increasing valuations, or increases in the rate of growth of valuations and thereby conclude you are not observing a bubble. It is actually more sound to say that what you are observing is consistent with our model of bubble dynamics.
Here is an example of the mix-up (from a bit above):
> The greatest values, and change in values comes just before the bubble bursts.
This seems to be talking about the mathematical model. Then you go on:
> In fact, tulips were most valuable, and those values were climbing at the highest rate before the tulip bubble burst.
That seems a jump to the modeled. And you speak of the modeled as if it was the model.
And by the way, if this graph [1] is correct, then what you say is also false. The valuations do not rise exponentially, but slow before the bubble bursts, very visibly. This is also true for the 2008 housing bubble in the local index [2].
I think that shows quite well that there are two layers. And there is the question of the fit between them. Having seen those two graphs I think I'd rather expect reduction in the rate of change of valuation shortly before the bubble bursts. That model appears to fit better what I'd expect in economical bubbles (but not in soap bubbles), namely a growing sense of risk that counteracts the fomo.
There is no issue with stating the model and an example of the modeled in the real world such as tulips. You are simply taking a favourable view on when the bubble "burst" occurs in order to see what you want. My point stands that valuations and there growth are not really distinguishing bubble/non-bubble. Thanks for proving me correct in that.
One could flip your post to say “AGI is coming” and be claiming the opposite, and it would be equally lacking insight. This is not “critical” to remember.
There are interesting and well thought out arguments for why the AGI is not coming with the current state of technology, dismissing those arguments as propaganda/clickbait is not warranted. Yannic is also an AI professional and expert, not one to be offhandedly dismissed because you don’t like the messaging.
I doing think that's fair to the person you replied to. At no time did they say they didn't like/dislike the message. Merely that there's a market for it, and thus, people may be biased.
Telling us all to remember that there's potential for bias isn't so bad. It's a hot button issue.
By this measure, considering the current capex all over the board, there is a lot more incentive in pushing the “AGI IS NEAR AND WE AINT READY” narrative than the opposite.
If AGI won’t come, as it’s highly probable, these companies are bust for billions and billions…
AGI is being able to learn from first principles, not being trained on trillions of examples. If you can start with priors and perform a demonstration that is not already a prior then you are intelligent. It’s the difference between the result and understanding the process that produces the result.
I just applied some stuff that’s been known for a long time in a different context. But let me give you a scenario to think about as a demonstration of how I mean.
Imagine we trained an AI on everything ever written, but the catch is we’ve restricted the training data to the year, let’s say, 400 BCE and earlier (Ignore the fact that most of what was written then is lost to us now and just pretend that’s not issue for our thought experiment) The AI is also programmed to seek new knowledge based off that starting knowledge.
Also pretend that this AI has an oracle it could talk to that would help the AI simulate experiments. So the AI could ask questions and get answers but only in a way that builds ever so slightly off what it already knows.
Making any progress at all in this experiment and discovering new knowledge is what we’re after and “new knowledge” would be defined as some r that’s demonstrated using some p and q as propositions, where r is neither p or q, and r is also correct in terms of 2025 knowledge.
If the AI, with the aid of the knowledge it started with and the help of the oracle to let it ask questions about the world and build off that knowledge, can ever arrive, or exceed, 2025 knowledge then it’s at least generally intelligent and equal to a human. Although the bar could maybe be even less.
It loses, however, if it never advances, gets stuck in a loop, or in some other sense can’t make progress.
This is intelligence: to proceed from things everyone agrees on and ask questions and formulate assertions that depend on propositions holding true, and in the process demonstrate new things that were not already part of common belief.
I don’t know how this experiment could be done in real life with an LLM for example but this is a story version of what I mean in my original comment.
Point taken! The question now turns to if I made the same mistake? I’d be grateful if you could demonstrate my mistake for me though, because I don’t see it.
Usually the burden of proof is on the one who is making a positive claim like: AGI is here or even AGI is coming.
The default position that does not need any more justification is the one that is skeptic or even agnostic to the claim that is made until proof is shown.
So when talking about evidence as a way to prove a claim: AGI is coming is the team that needs to provide this evidence. Someone saying AGI is not coming can add as many arguments or opinions as they like but it does not usually invite to such a high scrutiny as saying they need to provide evidence.
Using the very basic definition of AGI where "general" is just cross-domain, as in e.g. chemistry and law, the very first ChatGPT was already it. Not very smart one though.
"Modern" definitions that include non-intelligence related stuff like agency sound like goalpost moving, so it's unclear why would you want them.
Depends on definition of “cross-domain”. Can any of the current models be plugged in in an emulator of a human with equivalent senses (vision, hearing etc) and process those inputs in real time with the same speed of reaction, i.e. emulate human or animal intelligence in interactions with environment? That would be truly cross-domain.
Sensor and motor handling are important but not necessarily equal to general intelligence. A person suffering locked-in syndrome is still intelligent; an anemone responding to stimulus with movement is not.
I‘m not saying it’s equal, just pointing out that it’s a good indicator of processing capacity that is necessary to match human intelligence. We are aware of the environment and learn continuously from interactions with it. It should be expected that AGI can do the same.
Your case with locked-in syndrome is interesting. Can human brain develop intelligence in full absence of any senses? I doubt so.
It cannot. It doesn’t reason. Gambling and winning (1-10^N)*100% times is not the same as reasoning and providing accurate answers the same amount of times. If you reason about something, your errors fall into certain categories of fallacies often related to incomplete information. LLM hallucinations are easy to spot with reasoning, they are of statistical nature.
It is trained to practice chemistry and law. The reason it can do those because it's trained on an appreciable portion of all human output on both or those fields. If that's not training on them I don't know what is.
This sounds backwards to me. The fact that I can show you a law book and you will be able to pass a law exam is a consequence of you being "generally intelligent". If I show these books to something less intelligent like a cow, or less general, like a CNN, they won't be able to do that.
no, you're not understanding me. I'm not making any claim on whether they're any good at those subjects, merely that they are, objectively, trained on them
What is the relevance of this claim on its own if training on something alone is not sufficient for them to practice law, and you actually have to add cross-domain into picture for it to work, making it "general"?
I still disagree though, they are not trained to practice law. They are trained to remember law, but practice come from general training to follow instructions.
We have no reason to believe faster than light travel is possible. We know that AGI is possible, even if we don't know when or how. The comparison isn't apt.
OK, but your null hypothesis should always be a first or second degree linear projection.
"AI is getting better rapidly" is the current state of affairs. Arguing "AI is about to stop getting better" is the argument that requires strong evidence.
“AI is getting better rapidly” is a false premise. As AI is a large domain. There is no way to quanitify the better as compared to the entire domain. “Llms are improving rapidly during a short period of time where they gain popularity” is more accurate.
the same is true of solar eclipses, there are no partial eclipses, untill the very last instant of the moon covering the sun, it is far to bright to look at, and then the stars come out and the solar flares are visible, the birds sing there evening songs
and here I have told you of it, but at best it will be hint.
AI is worse, much worse, as we have our own inteligence, but cant even offer a hint of where that line is and how to cross it, where to go to see it
… is it? I hear people saying that. I see “improvement”: the art generally has the right number of fingers more often, the text looks like text, the code agents don’t write stuff that even the linter says is wrong.
But I still see the wrong number of fingers sometimes. I still see the chat bots count the wrong number of letters in a word. I still see agents invent libraries that don’t exist.
I don’t know what “rapid” is supposed to mean here. It feels like Achilles and the Tortoise and also has the energy costs of a nation-state.
Agreed there really isn’t any metrics that indicate this is true. Considering many models are still too complex to run locally. Llms are getting better for the corporations that sell access to them. Not necessarily for the people that use them.
Please, please seriously think back to your 2020 self, and think about whether your 2020 self would be surprised by what AI can do today.
You've frog-boiled yourself into timelines where "No WORLD SHAKING AI launches in the past 4 months" means "AI is frozen". In 4 months, you will be shocked if AI doesn't have a major improvement every 2 months. In 6 months, you will be shocked if it doesn't have a major update ever 1 month.
It's hard to see exponential curves while you're on it, I'm not trying to fault you here. But it's really important to stretch yourself to try.
I’m still surprised by what AI can do. It’s amazing. … but I still have to double check when it’s important that I get the right answer, I still have to review the code it writes, and I still am not sure there is actually enough business to cover what it will actually cost to run when it needs to pay for itself.
To be honest, I had the right idea back then... This technology has fundamental qualities that require it to provide inaccurate token predictions that are only statistically probable. They aren't even trying to change this situation other than trying to find more data to train, saying you have to keep adding layers of them, or are saying it is the user's responsibility.
There's been the obvious notion that digitizing the world's information is not enough and that hasn't changed.
I for one am quite surprised. Sometimes impressed. But also often frustrated. And occasionally disappointed. Sometimes worried about the negative follow-on effects. Working with current LLMs spans the whole gamut...
But for coding we are at the point where even the current level is quite useful. And as the tools/systems get better, the usefilness is going to increase quite a bit. Even if models improve slowly from this point on. It will impact the whole industry over the next years, and since software is eating the world, will impact many other industries as well.
Exponential? Perhaps in the same way as computers and Internet have been exponential - cost per X (say tokens) will probably go down exponentially the next years and decades, the same way cost per FLOP went down, on megabytes transferred. But those exponential gains did not results in exponential growth in productivity, or if so, the exponent is much much lower. And I suspect it will likely be the same for artificial intelligence.
What if I’ve not been impressed by giving a bunch of people a spam bot tuned to education materials? Am I frog boiled? Who cares about the actual advancement of this singular component if I was never impressed.
This is more interesting and deserves better discussion than we got from the previous title, which was derailed by the "AGI" bit, so I replaced the title with a representative sentence from the video.
(Edit: plus a question mark, as we sometimes do with contentious titles.)
>The era of boundary-breaking advancements is over
Maybe for LLMs but they are not the only possible algorithm. Only this week we had Genie 3 as in:
>The Surprising Leap in AI: How Genie 3’s World Model Redefines Synthetic Reality https://www.msn.com/en-us/news/technology/the-surprising-lea...
and:
>DeepMind thinks its new Genie 3 world model presents a stepping stone toward AGI https://techcrunch.com/2025/08/05/deepmind-thinks-genie-3-wo...
How different are world models from LLMs? I'm not in the AI space but follow it here. I always assumed they belonged to the same "family" of tech and were more similar than different.
But are they sufficiently different that stalling progress in one doesn't imply stalling progress in the other?
> How different are world models from LLMs?
Depends if you’re asking about real world models or synthetic AI world models.
One of them only exists in species with a long evolutionary history of survivorship (and death) over generations living in the world being modeled.
There’s a sense of “what it’s like to be” a thing. That’s still a big question mark in my mind, whether AI will ever have any sense of what it’s like to be human, any more than humans know what it’s like to be a bat or a dolphin.
You know what it’s like for the cool breeze to blow across your face on a nice day. You could try explaining that to a dolphin, assuming we can communicate one day, but they won’t know what it’s like from any amount of words. That seems like something in the area of neuralink or similar.
The world models are not really useful yet. So they are starting lower, compared to LLM. So they probably have some decent gains to make still, before it gets really hard (diminishing returns).
There are similarities with that one. From their website:
>It is comprised of a spatiotemporal video tokenizer, an autoregressive dynamics model, and a simple and scalable latent action model.
my point is more people can try different models and algorithms rather than having to stick to LLMs.
machine learning street talk has interview with the team
On the one hand, that isn't necessarily a problem. It can be just a useful algorithm for tool calling or whatever.
On the other hand, if you're telling your investors that AGI is about two years away, then you can only do that for a few years. Rumor has it that such claims were made? Hopefully no big investors actually believed that.
The real question to be asking is, based on current applications of LLMs, can one pay for the hardware to sustain it? The comparison to smartphones is apt; by the time we got to the "Samsung Galaxy" phase, where only incremental improvements were coming, the industry was making a profit on each phone sold. Are any of the big LLMs actually profitable yet? And if they are, do they have any way to keep the DeepSeeks of the world from taking it away?
What happens if you built your business on a service that turns out to be hugely expensive to run and not profitable?
>On the other hand, if you're telling your investors that AGI is about two years away, then you can only do that for a few years.
Musk has been doing this with autonomous driving since 2015. Machine learning has enough hype surrounding it that you have to embellish to keep up with every other company's ridiculous claims.
I doubt this was the main driver for the investors. People were buying Tesla even without it.
Whether there is hype or not, the laws of money remain the same. If you invest and don’t get expected returns, you will be eventually concerned and will do something about it.
Why are companies allowed to lie? I really can’t understand. If a person lies they lose credibility but it doesn’t apply to the rich and powerful.
Lying to investors is illegal, and investors have incentive and means to sue if they think they were defrauded. The problem is proving it. I'm sure a lot of founders genuinely believe AGI is about to appear out of thin air, so they're technically not lying. Even the cynical ones who say whatever they think investors want to hear are hard to catch in a lie. It's not really about being rich and powerful. That's just the unfortunate reality of rhetoric.
In addition to the other comments/answers to this, I would like to add that if you lie to your investors (in public), and they suspect you're lying but also think it will allow you to cash out before the lie becomes apparent, they may not care, especially if the lie is difficult to distinguish from pathological levels of optimism.
Predictions about the future and puffery are not illegal. Lying about facts are. Nobody knows how far away AGI is, everyone just has their own predictions.
It's not a crime to be wrong; it's only a crime to deliberately lie. And unless there's an email saying "haha we're lying to our investors", it's just not easy to prove.
so, without sarcasm: how many data centers is this non-happening worth? in other words, what justifies the huge spend?
I mean there are different definitions on what to call an AGI. Most of the time people don't specify which one they use.
For me an AGI would mean truly at least human level as in "this clearly has a consciousness paired with knowledge", a.k.a. a person. In that case, what do the investors expect? Some sort of slave market of virtual people to exploit?
Investors don't use this definition. For one because it contains something you cannot measure yet: consciousness.
How to find out if something has probably consciousness? Much less clearly? What is consciousness?
Being able to make arbitrary duplicates of slaves would be profitable, as long as the energy and compute is lower than salaries yeah
Do we have a reasonable definition for what intelligence is? Is it like defining porn, you just know it when you see it?
OpenAI defines AGI as a "highly autonomous system that outperforms humans at most economically valuable work" [0]. It may not be the most satisfying definition, but it is practical and a good goal to aim for if you are an AI company.
[0] https://openai.com/our-structure/
My personal definition is "The ability to form models from observations and extrapolate from them."
LLMs are great at forming models of language from observations of language and extrapolating language constructs from them. But to get general intelligence we're going to have to let an AI build their models from direct measurements of reality.
> LLMs are great at forming models of language
They really aren't even great at forming models of language. They are a single model of language. They don't build models, much less use those models. See, for example, ARC-AGI 1 and 2. They only performed ARC 1 decently [0] with additional training, and are failing miserably on ARC 2. That's not even getting to ARC 3.
[0] https://arcprize.org/blog/oai-o3-pub-breakthrough
> Note on "tuned": OpenAI shared they trained the o3 we tested on 75% of the Public Training set. They have not shared more details. We have not yet tested the ARC-untrained model to understand how much of the performance is due to ARC-AGI data.
... Clearly not able to reason about the problems without additional training. And no indication that the additional training didn't include some feature extraction, scaffolding, RLHF, etc created by human intelligence. Impressive that fine tuning can get >85%, but it's still additional human directed training and not self contained intelligence at the level of performance reported. The blog was very generous making the undefined "fine tuning" a footnote and praising the results as if they were directly from the model that would have cost > $65,000 to run.
Edit: to be clear, I understand LLMs are a huge leap forward in AI research and possibly the first models that can provide useful results across multiple domains without being retrained. But they're still not creating their own models, even of language.
LLMs have demonstrated that "intelligence" is a broad umbrella term that covers a variety of very different things.
Think about this story https://news.ycombinator.com/item?id=44845442
Med-Gemini is clearly intelligent, but equally clearly it is an inhuman intelligence with different failure modes from human intelligence.
If we say Med-Gemini is not intelligent, we will end up having to concede that actually it is intelligent. And the danger of this concession is that we will under-estimate how different it is from human intelligence and then get caught out by inhuman failures.
> Is it like defining porn
I guess when it comes to the definition of intelligence, just like porn, different people have different levels of tolerance.
One of my favorites is efficient cross domain maximization
Efficient, cross-domain optimization.
I believe that’s Eliezer Yudkowsky’s definition.
I did initially encounter it on LessWrong and modified it slightly according to my preference. Did he coin the term? There are a lot of ideas (not inappropriately) presented without attribution in that context.
As far as I know, it’s his own term.
mayhaps a prediction by an Artificial General Intelligence that is already here
Feels like we’re all just betting on the biggest “what if” in history.
To share my experience, 25 years ago I looked into AI and had inclinations to see what scaling compute would do. It took no time to find advisors who told me the whole program I had in mind could not gain ethics approval and was mathematically limited. The former road block is now lifted due to the fact that nobody cares about ethics any more, the latter seems to be the remaining hurdle.
The goal of economic is not to reach AGI. It would solve the problems we have with the current market, therefore would it make less money, then to just "chase" for the AGI. Shirky principle in a nutshell.
It is critical to remember that there is a market for people who say "AGI is not coming"
It doesn't matter whether they are lying. People want to hear it. It's comforting. So the market fills the void, and people get views and money for saying it.
Don't use the fact that people are saying it, as evidence that it is true.
You can remove the "not" and everything you wrote is just as true. If not more so.
It's not the AGI sceptics who are getting $500bn valuations.
Right, it’s exactly the opposite. What is the AI skeptic version of MIRI, for instance?
I guess the AI skeptic version of MIRI would be like, an organization that tries to anticipate possible future large problems that could arise from people anticipating an AGI that never arrives, but which people might believe has arrived, and proposes methods to attempt to prevent or mitigate those potential problems?
Can you say non-sequitor.
Oddly, in a bubble the highest valuations come just before the burst. This is obvious mathematical certainty that can be read by anyone viewing an exponential growth curve.
> obvious mathematical certainty
You mean like euclidean geometry?
I mean that as valuations rise before a bubble bursts, the curve provides that successive values grow, sometimes at an increasing rate. The greatest values, and change in values comes just before the bubble bursts. The point being that high valuations and increasing valuations are not very capable of distinguishing bubble/non-bubble. In fact, tulips were most valuable, and those values were climbing at the highest rate before the tulip bubble burst.
Look, I asked
>> obvious mathematical certainty
> You mean like euclidean geometry
To which you reply stuff about values and bubble bursts.
What you said may or may not be true. But it's hard for me to tell how it is related to what I asked.
I said that high valuation or valuations rising at an increasing rate is common to bubbles. This clearly refers back to the comment about the current high and rising valuations for AGI prospects.
I see that. I just have a hard time seeing how it is related to my question. Euclidean geometry was an "obvious mathematical certainty". Until it wasn't, and we learned that there are no "obvious mathematical certainties".
So I wanted to understand what you mean by this concept.
But we can leave it. It's not important. No harm meant.
There are mathematical certainties. In this instance we have that all exponentially growing bubbles will necessarily display the property that values grow and at an increasing rate just before the bubble bursts. This is so fundamental that I can't believe you don't see it. You can construct the growth curve out of this very fact alone. Therefore, the simple fact of growing values and increasing rates of growth in values is not determining bubble/non-bubble.
In other words, the statement "I know I am not chasing a bubble because valuations are high and growing at an increasing rate" is categorically false because this property is a common indicator of the exact opposite; that you are in an exponentially growing bubble.
Look, neither did I say there are no mathematical certainties, nor did I anywhere argue about the content of what you said.
The only thing I said is: there are probably no _obvious_ mathematical certainties in reply to what you said, as the history of mathematics shows with plenty of examples.
But it's already too much ink spilled for such a little difference.
I reckon now that you use words like "obvious" (or "categorically" in the last reply [1]) as markers of belief strength or something. That's fine, many people do it.
[1] related: https://news.ycombinator.com/item?id=36808333
> "Look, neither did I say there are no mathematical certainties, nor did I anywhere argue about the content of what you said
The only thing I said is: there are probably no _obvious_ mathematical certainties in reply to what you said, as the history of mathematics shows with plenty of examples."
There is nothing more obvious and certain mathematically than the fact that exponential growth of bubbles entails increasing valuations and increasing rates of growth in those valuations.
I use the word "obvious" quite properly as in "can be read from any graph", or "is contained within the structure or the model". I use categorical because the implied conclusion of the original comment is a categorical error.
There are people who disagree. Here is one compilation of theorems that are obvious and false:
https://math.stackexchange.com/questions/820686/obvious-theo...
This is not a theorem, it is a property of the model.
But I think I should be giving a reply also to the content of what you say. That's only fair after you took so much time explaining.
I think you conflate two questions in your statement: the question of what can be said about a function that is up to a point exponential and growing, and after that point decaying in an unspecified manner, but so that this point is a local maximum. And secondly the question if this is a good/the only/the best model for valuations in what is called an investment bubble that then bursts.
This conflation of model and modeled makes it hard for me to assess the argument, because I need to take these layers apart, and that involves guess work.
There are no layers and I am trying to help you in your difficulty. It is not sound to look around and observe increasing valuations, or increases in the rate of growth of valuations and thereby conclude you are not observing a bubble. It is actually more sound to say that what you are observing is consistent with our model of bubble dynamics.
Here is an example of the mix-up (from a bit above):
> The greatest values, and change in values comes just before the bubble bursts.
This seems to be talking about the mathematical model. Then you go on:
> In fact, tulips were most valuable, and those values were climbing at the highest rate before the tulip bubble burst.
That seems a jump to the modeled. And you speak of the modeled as if it was the model.
And by the way, if this graph [1] is correct, then what you say is also false. The valuations do not rise exponentially, but slow before the bubble bursts, very visibly. This is also true for the 2008 housing bubble in the local index [2].
I think that shows quite well that there are two layers. And there is the question of the fit between them. Having seen those two graphs I think I'd rather expect reduction in the rate of change of valuation shortly before the bubble bursts. That model appears to fit better what I'd expect in economical bubbles (but not in soap bubbles), namely a growing sense of risk that counteracts the fomo.
[1] image Tulip Price Index 1636-37 here https://www.historydefined.net/how-did-tulips-create-the-wor... [2] https://wolfstreet.com/wp-content/uploads/2021/07/US-Housing...
There is no issue with stating the model and an example of the modeled in the real world such as tulips. You are simply taking a favourable view on when the bubble "burst" occurs in order to see what you want. My point stands that valuations and there growth are not really distinguishing bubble/non-bubble. Thanks for proving me correct in that.
One could flip your post to say “AGI is coming” and be claiming the opposite, and it would be equally lacking insight. This is not “critical” to remember.
There are interesting and well thought out arguments for why the AGI is not coming with the current state of technology, dismissing those arguments as propaganda/clickbait is not warranted. Yannic is also an AI professional and expert, not one to be offhandedly dismissed because you don’t like the messaging.
I doing think that's fair to the person you replied to. At no time did they say they didn't like/dislike the message. Merely that there's a market for it, and thus, people may be biased.
Telling us all to remember that there's potential for bias isn't so bad. It's a hot button issue.
By this measure, considering the current capex all over the board, there is a lot more incentive in pushing the “AGI IS NEAR AND WE AINT READY” narrative than the opposite. If AGI won’t come, as it’s highly probable, these companies are bust for billions and billions…
Nobody even knows what AGI even is. This will most likely be defined by a corporation, not science. Due to obvious incentives.
It also could be like the Turing test.
There was no grand announcement of passing the Turing test or not. Instead the whole idea has faded in importance.
As the models get better, it wouldn't be shocking to me we get to a point that no one cares if the models are considered "AGI" or not.
We will be chasing some new vaguely defined concept.
AGI is being able to learn from first principles, not being trained on trillions of examples. If you can start with priors and perform a demonstration that is not already a prior then you are intelligent. It’s the difference between the result and understanding the process that produces the result.
Can you elaborate on the process by which you created this definition?
I just applied some stuff that’s been known for a long time in a different context. But let me give you a scenario to think about as a demonstration of how I mean.
Imagine we trained an AI on everything ever written, but the catch is we’ve restricted the training data to the year, let’s say, 400 BCE and earlier (Ignore the fact that most of what was written then is lost to us now and just pretend that’s not issue for our thought experiment) The AI is also programmed to seek new knowledge based off that starting knowledge.
Also pretend that this AI has an oracle it could talk to that would help the AI simulate experiments. So the AI could ask questions and get answers but only in a way that builds ever so slightly off what it already knows.
Making any progress at all in this experiment and discovering new knowledge is what we’re after and “new knowledge” would be defined as some r that’s demonstrated using some p and q as propositions, where r is neither p or q, and r is also correct in terms of 2025 knowledge.
If the AI, with the aid of the knowledge it started with and the help of the oracle to let it ask questions about the world and build off that knowledge, can ever arrive, or exceed, 2025 knowledge then it’s at least generally intelligent and equal to a human. Although the bar could maybe be even less.
It loses, however, if it never advances, gets stuck in a loop, or in some other sense can’t make progress.
This is intelligence: to proceed from things everyone agrees on and ask questions and formulate assertions that depend on propositions holding true, and in the process demonstrate new things that were not already part of common belief.
I don’t know how this experiment could be done in real life with an LLM for example but this is a story version of what I mean in my original comment.
> I just applied some stuff that’s been known for a long time in a different context.
I tried this, and screamed "Goal!!!!" on a golf course when the ball went in the hole. Did not work as expected. Lost my membership.
But maybe I applied it wrong.
Point taken! The question now turns to if I made the same mistake? I’d be grateful if you could demonstrate my mistake for me though, because I don’t see it.
Not at all. I was just implying that the process definition was a bit vague. Next time I'll state it clearly.
Understood. I couldn’t explain everything about the process because it would be too long, but it’s basically Aristotle’s Organon.
Wake me up when there is a formal definition of human sentience that everyone agrees with.
Usually the burden of proof is on the one who is making a positive claim like: AGI is here or even AGI is coming.
The default position that does not need any more justification is the one that is skeptic or even agnostic to the claim that is made until proof is shown.
So when talking about evidence as a way to prove a claim: AGI is coming is the team that needs to provide this evidence. Someone saying AGI is not coming can add as many arguments or opinions as they like but it does not usually invite to such a high scrutiny as saying they need to provide evidence.
Using the very basic definition of AGI where "general" is just cross-domain, as in e.g. chemistry and law, the very first ChatGPT was already it. Not very smart one though.
"Modern" definitions that include non-intelligence related stuff like agency sound like goalpost moving, so it's unclear why would you want them.
Depends on definition of “cross-domain”. Can any of the current models be plugged in in an emulator of a human with equivalent senses (vision, hearing etc) and process those inputs in real time with the same speed of reaction, i.e. emulate human or animal intelligence in interactions with environment? That would be truly cross-domain.
Sensor and motor handling are important but not necessarily equal to general intelligence. A person suffering locked-in syndrome is still intelligent; an anemone responding to stimulus with movement is not.
I‘m not saying it’s equal, just pointing out that it’s a good indicator of processing capacity that is necessary to match human intelligence. We are aware of the environment and learn continuously from interactions with it. It should be expected that AGI can do the same.
Your case with locked-in syndrome is interesting. Can human brain develop intelligence in full absence of any senses? I doubt so.
I defined cross-domain with an example. ChatGPT is not trained to practice chemistry and law, yet it can do both. It is cross-domain.
You can make it stronger at being cross domain, but it satisfies the minimum requirement.
It cannot. It doesn’t reason. Gambling and winning (1-10^N)*100% times is not the same as reasoning and providing accurate answers the same amount of times. If you reason about something, your errors fall into certain categories of fallacies often related to incomplete information. LLM hallucinations are easy to spot with reasoning, they are of statistical nature.
> (1-10^N)*100% times
It is trained to practice chemistry and law. The reason it can do those because it's trained on an appreciable portion of all human output on both or those fields. If that's not training on them I don't know what is.
> It is trained to practice chemistry and law
No, it is not. It is trained to predict next token, and it is trained to follow user instructions.
I mean, it a very literal sense, it is. Chemistry and law textbooks are in the training corpus. It is trained on these
This sounds backwards to me. The fact that I can show you a law book and you will be able to pass a law exam is a consequence of you being "generally intelligent". If I show these books to something less intelligent like a cow, or less general, like a CNN, they won't be able to do that.
no, you're not understanding me. I'm not making any claim on whether they're any good at those subjects, merely that they are, objectively, trained on them
What is the relevance of this claim on its own if training on something alone is not sufficient for them to practice law, and you actually have to add cross-domain into picture for it to work, making it "general"?
I still disagree though, they are not trained to practice law. They are trained to remember law, but practice come from general training to follow instructions.
Faster than light travel is not coming.
Given that AGI does not exist, “AGI is not coming” is the status quo until someone disproves it.
We have no reason to believe faster than light travel is possible. We know that AGI is possible, even if we don't know when or how. The comparison isn't apt.
Waiting for Agi-dot…
The inverse can be true too: Just because people ARE saying that Agi is coming, isn’t evidence that it is true.
OK, but your null hypothesis should always be a first or second degree linear projection.
"AI is getting better rapidly" is the current state of affairs. Arguing "AI is about to stop getting better" is the argument that requires strong evidence.
“AI is getting better rapidly” is a false premise. As AI is a large domain. There is no way to quanitify the better as compared to the entire domain. “Llms are improving rapidly during a short period of time where they gain popularity” is more accurate.
Llms getting better != a path to AGI.
the same is true of solar eclipses, there are no partial eclipses, untill the very last instant of the moon covering the sun, it is far to bright to look at, and then the stars come out and the solar flares are visible, the birds sing there evening songs and here I have told you of it, but at best it will be hint. AI is worse, much worse, as we have our own inteligence, but cant even offer a hint of where that line is and how to cross it, where to go to see it
> "AI is getting better rapidly"
… is it? I hear people saying that. I see “improvement”: the art generally has the right number of fingers more often, the text looks like text, the code agents don’t write stuff that even the linter says is wrong.
But I still see the wrong number of fingers sometimes. I still see the chat bots count the wrong number of letters in a word. I still see agents invent libraries that don’t exist.
I don’t know what “rapid” is supposed to mean here. It feels like Achilles and the Tortoise and also has the energy costs of a nation-state.
Agreed there really isn’t any metrics that indicate this is true. Considering many models are still too complex to run locally. Llms are getting better for the corporations that sell access to them. Not necessarily for the people that use them.
Compare Altman outlandish claims about GPT-5 and the reality of this update. Do you think they square out in any reasonable way?
Please, please seriously think back to your 2020 self, and think about whether your 2020 self would be surprised by what AI can do today.
You've frog-boiled yourself into timelines where "No WORLD SHAKING AI launches in the past 4 months" means "AI is frozen". In 4 months, you will be shocked if AI doesn't have a major improvement every 2 months. In 6 months, you will be shocked if it doesn't have a major update ever 1 month.
It's hard to see exponential curves while you're on it, I'm not trying to fault you here. But it's really important to stretch yourself to try.
I’m still surprised by what AI can do. It’s amazing. … but I still have to double check when it’s important that I get the right answer, I still have to review the code it writes, and I still am not sure there is actually enough business to cover what it will actually cost to run when it needs to pay for itself.
To be honest, I had the right idea back then... This technology has fundamental qualities that require it to provide inaccurate token predictions that are only statistically probable. They aren't even trying to change this situation other than trying to find more data to train, saying you have to keep adding layers of them, or are saying it is the user's responsibility.
There's been the obvious notion that digitizing the world's information is not enough and that hasn't changed.
I for one am quite surprised. Sometimes impressed. But also often frustrated. And occasionally disappointed. Sometimes worried about the negative follow-on effects. Working with current LLMs spans the whole gamut... But for coding we are at the point where even the current level is quite useful. And as the tools/systems get better, the usefilness is going to increase quite a bit. Even if models improve slowly from this point on. It will impact the whole industry over the next years, and since software is eating the world, will impact many other industries as well. Exponential? Perhaps in the same way as computers and Internet have been exponential - cost per X (say tokens) will probably go down exponentially the next years and decades, the same way cost per FLOP went down, on megabytes transferred. But those exponential gains did not results in exponential growth in productivity, or if so, the exponent is much much lower. And I suspect it will likely be the same for artificial intelligence.
What if I’ve not been impressed by giving a bunch of people a spam bot tuned to education materials? Am I frog boiled? Who cares about the actual advancement of this singular component if I was never impressed.
You assume everyone is “impressed”.
What products are people building on not-AGI?