> If you’re under the impression that these models are “glorified autocomplete” or that progress is slowing down, I need to urge you: stop thinking that. The models are very intelligent and capable, they are getting better at a fast clip. I can cite measurable and impressive progress over just the past five months on specific types of problem I’ve asked them to look at. [...]
> On the other hand: if you think that models are super-intelligent or that AGI is already here, you should also stop thinking that. Working with these tools is like swimming in a pond where the ground drops off sharply. One minute you’re wading comfortably and there’s support under your feet. Then suddenly you cross a specific line, and you’re back to swimming on your own.
There is a vast ocean of human knowledge, far beyond the capacity of any human brain, even within specialised fields.
Books helped "plug the gaps" in our knowledge, increasing the scope that a single human mind can encompass.
Web search engines did the same thing, but more and faster.
LLMs are like search engines on steroids, essentially a research librarian that operates at 1,000x human speed and can "in context" locate relevant information, adapting it to fit the hole it needs to go into as well.
It feels less like discovering new theorems, but instead having direct access to all theorems, which is hugely valuable in itself.
I.e.: the recent counterexamples to open conjectures has largely been about the AIs "trawling through all the things" and scraping together every bit of human-generated knowledge ever produced that is relevant to the conjecture.
Conversely, in the past, we had to "make do" with sub-standard solutions where the problem had been solved, but finding every relevant solution in the ocean of knowledge was prohibitively time consuming.
In some sense, LLMs will "raise the floor" in what is considered the minimum level of quality of a solution, where even throwaway / toy designs will now start applying every bit of accumulated wisdom instead of just some of it.
That is a good analogy. But the limitations of LLMs seem to be where they lack some information they tend to hallucinate/invent.
The rare benchmarks that measure "knowing what the model knows it doesn't know" show us there are only a couple models like Opus that are good in that field.
I think this is something that receives not enough attention from researchers.
I feel like there has been a ton of noise about this, but frankly, no one has actually defined what AGI means. I feel like the goal post is constantly shifting.
Take for example Humanity's Last Exam. It is so broad and complex that while an individual in a specific field might be able to answer their specific area of questions, they certainly would not be able to achieve >50% on the total question set.
There is this idea that AI has to be perfect to be intelligent - but we consider Humans intelligent and they are not even close. So is it the ability to generate novel ideas? Prove theorems? Pass tests?
I am not arguing that rote memorization is intelligence, or that we have achieved it, but does anyone know what AGI actually.. is?
> I am not arguing that rote memorization is intelligence, or that we have achieved it, but does anyone know what AGI actually.. is?
I would argue that a core part of intelligence is being able to handle uncertainty. That + planning probably explains most of the evolutionary pressure for making our brains bigger. But if this is important to intelligence, chess bots in the 80s were more intelligent than their later counter-parts, which could simply remove uncertainty through rote memorization. Maybe the All-Knowing is a compete dud, no reasoning capabilities at all, just an extremely efficient, infinite lookup table of all facts.
Terms like intelligence and consciousness often just seem overloaded with meaning, and when discussing things concretely, we quickly switch to more specific terms like reasoning.
"The intelligence of a system is a measure of its skill-acquisition efficiency over a scope of tasks, with respect to priors, experience, and generalization difficulty."
Subtextually, when people discuss AGI as a threshold, they're talking about superintelligence, which I'd roughly define as reasoning (not just applying algorithmic search) at the speed with which computers do lookups and computations, with large-scale data set contexts.
Right now, models are doing well-defined knowledge work tasks, applying relatively well-worn patterns (but with the thoroughness of a computer, which yields interesting new results). AGI is us breaking the threshold of "doing stuff people already do to identify and solve problems".
It’s always been nebulous but that was fine because we were so incredibly far away from it. We never really planned to be close to it figuring out the edge. Some have it at human or beyond for all tasks, but then rarely touch on “one human” or “all humans”.
Personally having been in AI since before deep nets, systems have been incredibly narrow for decades.
Classifiers on images were battling with ten classes in 2010. Imagenet had 1k classes and people were getting half of the things wrong then and that was frankly amazing at the time.
And they only did images, only to known classes, only with very specific inputs.
Text classifiers only did a few classes usually and mostly threw all the words together.
The most advanced things I saw in the late 2000s were struggling so much to make general systems that the most general ones were still incredibly limited and bad at those things (we had a robot learning to play games that you
showed it). Things like asking a thing for a book and having it parse the sentence, identify what was needed, that it didn’t know where it was but that was knowledge another human had and asking them - that was impressive yet also limited to very small sets of interactions.
The idea of a machine getting sarcasm, even if mostly built for it, was wild.
General meant capable of a broad range of tasks without retraining.
To me we have agi. It’s general, and it’s good enough to be useful.
I think it's just that we have hard time pinning down what is it that intelligence factor that isn't well-covered by LLMs. Just like we've had trouble with discerning human intelligence in the past.
It turns out that a savant with all the knowledge isn't "it".
There's a specific singularity theory of AI that is very popular. Eliezer Yudkowsky helped popularize it among the Bay Area "rationalist" community, and it has this idea that AGI necessarily implies a self improving system that will quickly become a paperclip maximizer or other such dystopian or utopian world changing intelligence.
By that singularity definition, we're probably nowhere near AGI, but if we define it as something that is as good at text/information manipulation as the 50th percentile human? I think we're already there.
> ...a hypothetical type of artificial intelligence that matches or surpasses human capabilities across virtually all cognitive tasks.
You can argue that paperclip maximising is an inevitable consequence of that (and the huggingface breach is interesting from that point of view) but it's not fundamental to the definition.
The question then is what "surpasses human capabilities" means and we're there in some niches but not all, and not across many models.
I think its fair to blame the AI labs for constantly hyping up the intelligence in a consiousness-related way where the model becomes "just like a human". Of course that is bs in terms that we have no clue on consiousness of the underlying models or even if its possible to do so.
But a purist defintion of AGI does not require consiousness as a part of it but can be seen as a benchmark metric. Sam Altman recently said the term doesn't really matter. And its true, even if AGI is achieved in the sense that a model can excel at all tasks at par or better than a human, then it is very useful but not as scary as a living, self-serving AI system like Skynet.
We'll probably be able to tell if and when Yudkowsky's AGI arrives.
Once AI can improve itself, frontier labs will no longer need human developers. So we might see a massive layoff of top talent and a dramatic increase in product quality at the same time. This usually doesn't happen in human businesses. It is also very much against the interest of anyone who is already at the top of the pay table at those labs.
Layoffs would be a poor indicator that recursive self-improvement had occurred as you need people with knowledge around domain/layer the work is done on.
Also, if the lab truly has a self-improving superintelligence, the cost of retaining staff at any level would be a rounding error relative to its operating costs and the value the system creates.
There would be little economic pressure to fire them immediately, especially while they remain useful for oversight, interpretation, risk management or simply as "interface" to the rest of the world etc.
If anything, they would probably hire more people to pursue more opportunities in parallel.
> Layoffs would be a poor indicator that recursive self-improvement had occurred as you need people with knowledge around domain/layer the work is done on.
This is why I keep saying we can't all agree even on a single letter of "A", "G", and "I".
Before ChatGPT, I would have said "obviously a generally intelligent system can do all the things". While LLMs are much more general than AI before them, the quality of their performance in all the things is distributed in a very un-human-like way.
Some fast-moving optimiser can be a threat well before it stops needing any humans for part of their labour. Cancer and viruses are examples of this: they're the same category of thing as a paperclip optimiser, but for biology instead of manufacturing office supplies.
But some others will argue LLM-spikey isn't "AGI", they'll demand something which reaches the performance of the best human (or the mean human, or the mean domain expert, because we can't agree on "I"), and a standard of "≥ best human" would mean that no, you don't need "people with knowledge around domain/layer the work is done on".
AI is already improving itself. Most / all of the coding harnesses are AI-written.
And yet… there are still people telling AI how to improve itself.
IMO there will always be a level of abstraction at which AI needs guidance. Perhaps ASI means it decides everything on its own, but I don’t think so. Genius humans often excel at the how but not the why, or even the what. So far there’s no indication that AI is different.
> AI is already improving itself. Most / all of the coding harnesses are AI-written.
AI has not improved the network topology much yet. The next (and possibly 'last') big thing is enabling AI to come up with something as impactful as the transformer architecture.
It’s a fair point, but I’d argue “AI” is the whole stack, not just model weights. And AI is absolutely improving the whole stack in ways that make it smarter (from a consumer’s perspective).
True, but that is clearly not where the biggest potential gains lie. The number of different topologies we've tried for ANNs is miniscule compared to the number of biological neural networks evolution has tried. The latter are also far more intricately organized.
Honestly, when it comes to fundamental ANN topology improvement we've only just gotten started.
What you describe would still be limited with so called AGI due to the existing organizational, legal and social structures around basic things like responsibility. We could have scifi level robots available today and these non-tech issues would still need to be resolved for a company to just fire all it's employees.
> I feel like there has been a ton of noise about this, but frankly, no one has actually defined what AGI means. I feel like the goal post is constantly shifting.
I'd say LLMs have shown us the opposite problem: there were many different definitions whose differences we'd previously been able to ignore. We don't all agree even on a single letter of "A", "G", and "I".
I'm also getting irritated with the “glorified autocomplete” comments. Since nobody can post such comments and also use the tools I'm using, I'm wondering if the phenomenon is due to people only having experience with the free version of whatever it is they're trying to use?
The “glorified autocomplete” framing isn’t to take literally. It’s a way to remove the mystic and whole anthropomorphization of AI. It’s saying they aren’t sentient or entities we are interacting with, even if that’s how the output presents itself. Instead they are “just” stochastic models
Some people use it to demystify, but a whole lot of people seem to be using it to dismiss the technology entirely.
Personally I like to remind people that these things are next-token predictors, but then emphasize how truly astonishing the results we can get out of sufficiently advanced next-token predictors are.
The "next-token predictor" framing is also a bit shaky. It's an accurate description of pre-training, where next-token prediction is a useful learning objective to force the model to learn higher-level representations. It's wildly misleading for a model put through an RL post-training campaign. The tokens it "predicts" aren't sampled from any naturally occurring distribution; the model's output is the result of an optimisation process that rewarded behaviour that was useful, and that's fundamentally different.
> The tokens it "predicts" aren't sampled from any naturally occurring distribution; the model's output is the result of an optimisation process that rewarded behaviour that was useful, and that's fundamentally different.
"Surprisingly, we find that the current training setup does not elicit fundamentally new reasoning patterns. While RLVR-trained models outperform their base models at small k (e.g., k = 1), the base models achieve a higher pass@k score when k is large. Coverage and perplexity analyses show that the observed reasoning abilities originate from and are bounded by the base model. "
This paper is less dramatic than you think it is and really just re-explains what RLVR does.
Let's stipulate that what pretraining does is train next token prediction over a gigantic corpus. You can then sample from this distribution repeatedly (cf the Large Language Monkeys paper) and count how often it passes some deterministic verifier.
What GRPO-style RLVR does is precisely this, but then reward the trajectories which passed the verifier. These distributions are _by construction_ within the accessible output space of the pretrained model; you're reweighting the distribution so that pass@k goes up, because that's (for applications like programming) very useful. RLVR is about making sampling more efficient; the only new information being added to the system is the presence of the verifier, and note that you only get a reward when the verifier passes, so there's essentially no mechanism for "teaching new facts" here.
Ooh. This looks like an interesting paper and there were a couple of things in the intro that I found counter-intuitive. It'll take me a while to digest the whole thing.
> Coverage and perplexity analyses show that the observed reasoning abilities originate from and are bounded by the base model
On the face of it this seems unsurprising given the policy gradient term directly minimises this difference.
I don't have a good feel for how the output of an RLVR-trained model concretely differs from the base model. My guess would be there are a fairly small number of "forks" where the training creates a token flip that sends the model down a more useful path.
The fact that the straight paths between the forks resemble the base model would again be unsurprising since (a) those are exactly the right context to continue to elicit more output that's relevant to solving the problem (so not penalised by RLVR), and (b) preservation drops naturally out of the policy gradient term you add to limit catastrophic forgetting in the base model.
Low perplexity could be explained by the relative sparsity of the forks in the output stream, and/or by forks already having high entropy in the base model. That also aligns with the pass-at-high-k: yes it's doing more exploration without training but it's a bit of a monkeys-on-typewriters situation.
Lack of novelty is readily explained by the fact that you need some nonzero pass rate in the base model to actually get some useful training signal from RLVR. That's a limitation of contemporary RLVR techniques, not a limitation on post-training in general.
I think there's room in that forks-and-straights characterisation for the RLVR'd model to be doing something that looks a lot like computation, while having low perplexity vs the base model. I don't see anything in my admittedly incredibly shallow skim of the paper that refutes that.
VibeThinker 3B is highly problematic for the thesis expressed in that paper. There is no possible way it could derive its reasoning capabilities from a bag of 3B parameters alone.
Right, but it's still useful to think of these models in terms of next-tokens because it helps explain that they look at every token that came before and use that to put out the next one.
You can get into RL as part of explaining why it's so unnervingly good at picking a next token.
That's true. The fact that an LLM is a pure function of (all previous tokens) -> (next token), with internal state like KV cache only existing for optimisation purposes, is pretty mind-blowing.
I guess it was more the "predictor" part I had issue with. There's a tendency to reach for statistical or probabilistic terminology to describe things that aren't usefully understood in those terms. For example in the "Speed Always Wins" LLM technical survey (https://arxiv.org/pdf/2508.09834):
> The gate is a crucial component to bring sparsity in MoE models. For a batch of input token representations X ∈ RT×D, the gate function G determines the probabilities of dispatching token xi to each expert e
...which is nonsense: the gate simply, directly, selects the experts. There's nothing probabilistic about it.
> the model's output is the result of an optimisation process that rewarded behaviour that was useful, and that's fundamentally different
I'm not understanding, can you explain this more? How does it become more than a next token predictor? Isn't the post-training simply altering the sampled distribution? And isn't that distribution naturally occurring? It's the distribution of "useful" next token?
IMO that doesn’t sound so much like prediction any more.
It’d be prediction if it’s “predict what would come next in this text sampled from distribution X”.
But what’s it predicting if we’re looking for new useful outputs? It’s finding a distribution that’s useful, and generating tokens, but it’s not predicting what comes next in a known sequence.
> predict what would come next in this text sampled from distribution X
It tries to learn the distribution of "useful" results either through verified rewards or human feedback. Then it encodes that in the network. When you run inference later, it samples or selects from that distribution.
Maybe it's a matter of interpretation. It's not predicting the next token based purely on the training corpus's distribution anymore, the RL process fine tunes that distribution so it predicts the next token that is closer to what was rewarded during RL. But as I see it, it's still predicting the next token, just from a reenforcement learned distribution instead of one found in a corpus of data.
I'm curious, what are you hoping to convey by reminding people that LLMs are next-token predictors? They are, of course, but most people without an AI background won't fully understand what that means, so I assume you're using it at least partly as a proxy for something else.
I think understanding how this stuff works is really important. For technical people it gives them a useful starting point for understanding it all. For less technical people it's crucial to help them understand that it's not some weird new magical science-fiction AI - it's still computer programs that turn text into numbers and do stuff with the numbers and turn those back into text.
It's harder to believe something is conscious or threatening to achieve word domination once you understand that it's a machine that statistically figures out which word should come next.
I don't think the next-token-predictor thing should increase anyone's confidence that LLMs aren't conscious or can't escape the control of their operators. A very closely analogous argument would "prove" that humans aren't conscious or can't do [insert task here] either. (No, I'm not saying that any of this is true of today's LLMs, I'm saying this particular argument doesn't work.)
You can say that for any argument regarding consciousness, because we don’t have an actual, all encompassing definition of what consciousness is. In general I don’t think comparison with humans makes much sense, we should be able to discuss LLMs without always falling back to “but what about humans” (sorry for the caricature)
Shouldn't that imply that agnosticism is the proper view, rather than asserting that something is impossible on a next-token-predictor architecture?
(Note: I don't actually think the consciousness question is the most important one in the near term. Where I think this line of reasoning gets really dangerous is when people use it to assert that LLMs can't or won't engage in certain behaviors no matter much they advance; this doesn't have anything to do with consciousness.)
There are some vague notions about consciousness emerging from complexity that some people may advance as a nuance to your point, although that just makes rocks “minimally conscious”, not necessarily unconscious. If you take the view that the entirety of our consciousness’s comes from purely classical interactions (electrical and chemical), then thats not a difficult conclusion to arrive at, but the notion that consciousness can be derived from deterministic computation does not pass the sniff test imo.
As a side note, that is why I find the idea that the brain is a quantum-classical hybrid computer appealing. And following the research developments is very interesting, to say the least.
I think you can reliably assert that X != Y without having a complete definition of Y, as long as you can identify at least one property or condition that Y possesses which X violates.
So for consciousness and LLMs it could be Qualia, lack of semantic understanding, lack of continuity in time, lack of a high degree of integrated causal feedback, etc.
Or perhaps those are just features of human consciousness but not integral to consciousness as a whole. To me this then implies panpsychism to some degree, which I'm alright with too.
I would think that qualia is the element of that list that actually matters here- "Is there a way that it feels to be an LLM?" is, I expect, the underlying question being asked within "Is an LLM conscious?". If LLMs could only experience the timeless, nonlocalized color blue, they'd still be conscious. The presence of positive and negative valence qualia is additionally relevant if someone is getting at whether they have moral worth, but it's secondary.
But qualia are not directly measurable and the rest of the list only matters if those features are necessary for qualia, which we can't decide without such measurements or at least a strong theoretical model.
While you’re active in this thread, I just want to say thank you for all your writing, you’re such a reliable source of sanity in that crazy new world :)
It's harder to believe something is conscious or threatening to achieve word domination once you understand that it's a machine that statistically figures out which word should come next.
The problem is, these models challenge our definition of "consciousness." Or at least they point out how hopelessly-inadequate our thinking on the subject is. Some people really, really don't like having their personal definition of consciousness challenged.
The correct response to "So what, it's just a next-token predictor" isn't a long dissertation on RLHF, training architectures, scaling laws and whatever, but rather to turn around and respond, "Sure, and how is that different from what we do?"
LLMs self-evidently have no experience when not inferencing, seems like the most consequential difference to me. If we are next token predictors then we are next token predictors that are inferencing at every waking moment and arguably much of our sleeping moments too; when we train an LLM that can inference as much as we do and remain coherent then I'll be more worried about whether it might have experience.
> It's harder to believe something is conscious or threatening to achieve word domination once you understand that it's a machine that statistically figures out which word should come next.
At the risk of sounding overly flippant, all world domination has been achieved by some person(s) figuring out which word should come next. Words quite literally = action when it comes to LLM’s with tools access
> At the risk of seeming X, statement that overwhelmingly demonstrates X-ness.
It's exhausting to even consider where to begin addressing the assertion that good leadership is just predicting the next word to say. Especially considering the corpus available to most great leaders in history was extremely small. To think Hannibal's military campaigns were just because he'd read like ten books in his life and could accurately forecast effective rhetoric is...indescribably divorced from reality.
I think this is smart, it seems to me that the best way to use these models its to approach them as a next-token predictor instead of an intelligent entity. That's how I've gotten the best results from them and allows me to avoid some of the pitfalls people fall into by anthropromising them.
Exactly. They ARE "glorified autocomplete" in an ontological sense. That says nothing about capability or outcome. The people who come out swinging against that characterization usually ignore the whole ontological argument (which is...the entire point) and go after an outcome-based strawman.
Depends. "Most" implies majority, and the majority of people are using these tools not for programming but in contexts where ontology is more relevant than capability (not that capability is irrelevant, but most people care, or are tricked into caring, far more about the former).
I don't know about "most", but there are a lot of people treating it as somewhere between "magic oracle" and "new friend"
Not to mention people more worried about whether the AI is motivated to hurt us than what human motivations can do with something that can autocomplete its way through every possible attack vector of cryptographic systems most of use would prefer remain secure.
(tbf I think the "glorified autocomplete" still works surprisingly well for programming outcomes too. Autocomplete [and fuzzy search of reference material] actually is useful and often right and certainly can save time even when it's only suggesting the rest of the variable name. But you might not want to commit everything it suggests...)
My favorite analogy for LLMs is "an intellectual chain saw". It does one thing, well, but that isn't "cut wood". It's "cut". What it cuts will happily include limbs.
But... These are also literally glorified autocomplete. Autoregressive language models are all autocomplete. It turns out that advanced enough autocomplete can do interesting stuff.
There was an interview I watched with Cory Doctorow just a few weeks ago, where someone asked him, "What kind of evidence could be presented to you to make you believe that LLMs were now ready to replace people at jobs entirely?" And he basically responded that he didn't think it was possible they ever would, because they're not really thinking, they just "predict the next token". This in spite of him actually having first-hand experience with people getting loads of value out of Claude Code and GPT Codex.
I generally like it still. It describes their failure modes pretty well, and in a way that most people already recognize. They're incomparably more complex, of course, but they are not intelligent and they are very much repeating what they've seen without any capability for factual accuracy.
Practically every other attempt at describing them leans too technical and unfamiliar (stochastic parrot) or too anthropomorphic (even describing them as "not like a human" gets people thinking in terms of humans, like how if I mention that your tongue is in your mouth all the time, using up almost all of the room, feeling your teeth and tasting itself, you're now uncomfortably aware of it and the numerous bumps on the surface).
You need to work from a reference that has both a shared understanding, and does not lead to problematic "if X has Y, and Z is like X, then Z has Y" seemingly-logical derived beliefs. "Spicy autocomplete" is a fairly safe starting point in both ways.
"Repeating what they've seen" could be, depending on their generalization abilities, something like low-n-gram Markov chain, repeating of surface-level speech patterns, repeating reasoning patterns, repeating mind-theory-level patterns, repeating self-correction patterns and so on.
And they fall somewhere between "surface-level speech patterns" and "repeating reasoning patterns" imo, yeah. They do indeed learn higher-order patterns than a Markov chain... but they'll blatantly misapply reasoning patterns and self-contradict confidently, just like you see with human bullshitters who don't understand one bit of what they're doing.
They sound fairly human, until you notice the patterns. They sound like they're thinking, until you pay attention.
It’s such willful blindness though. Why deny that there is a huge gulf between autocomplete and sentience? It’s like saying cars are glorified bicycles because they’re sure not airplanes.
Which obviously raises the question of if we are just glorified autocomplete (or glorified pattern matching). After all, we evolved from things just trying to recognize the patterns for what will hurt us and what will cause pleasure.
Let's say I'm one of such people. How would you convince me that your comment is any different from the ones we've been getting since roughly GPT-3 claiming that AGI is finally here and that, while the previous models were bad, this time they'll live up to the hype?
For me it's more an expression of how much mileage you can get out of "glorified", how many tasks devolve down to "if you model language accurately enough, look what drops out" because it turns out that to model language you need to model how the world works. I don't use it to minimise the capabilities at all.
The models themselves are indeed glorified autocomplete in terms of what they actually do (with things like agentic coding harnesses being required as a wrapper around them to make that internal autocomplete something more useful). Many people use this fact to critique LLMs, but many other common instances of people pointing out LLMs' apparent lack of intelligence actually come from people not understanding that the model is a glorified autocomplete underneath whatever interface people access them through, and the interface isn't providing the underlying model all the information they assume it would, making it seem less intelligent than it actually is.
> both outputs of Claude Mythos, their (still) unreleased advanced model
That sentence gives the impression that Mythos might be released in the future. That's clearly not going to happen - it's already "released" in as much as selected, trusted partners can access it, and the rest of us get it in the form of Fable - which is Mythos but with filters that downgrade you if you try to use it for anything even remotely related to cybersecurity or biology.
(The other day Fable 5 downgraded me to Opus after I asked it to explain the difference between tusks and teeth.)
Got downgraded from Fable to Opus after asking about the Great Oxidation Event [1], presumably because it started thinking about cyanobacteria, which made the monitor afraid I was trying to make a biological weapon or something.
because as OP of the article points out, it's not a guaranteed hit with these models no matter how capable. they are stochastic (but not parrots), humans are too (but in a different way), and so there is a chance that the answer is wrong. If I wanted the most accurate answer I'd be confident in believing, I'd ask the most advanced model.
Now obviously you can and should retort with hallucination and confabulation rates from external and Ant's own reports per model (pretty sure more advanced models are good at lying better, not less) instead of going with dumb "more expensive more accurate" mental model, but general principle stands for me still AFAIK.
it's not exactly rational I admit, but if I'm going to base my own work and reasoning from an LLM I'm going with the best available. This seems to trip up most normies because they are too lazy or too greedy to pay up for premium access and see for themselves why most of us are both awed and afraid. Generally, I'm too biased and too deep in ML/DL cargo cult (been in it since 2016) to know if the skepticism and disdain for such usage is warranted.
In general, I think the tools are broadly toxic in a Dune-sense of making me think less for myself, because just as any HN-poster knows coding and doing mundane low-level stuff is necessary the same way doing stretches is necessary before any workout. The process itself is what keeps your brain strong and its gradients from veering into overfitting. I'm not overly bullish on the whole reaching for the stars ending with these things. Paradoxically, you using them eventually hobbles both you and the model, because you become dumber and then you bottleneck their ability to self-direct (broadly true for next Mythos/GPT-7).
Sorry for a long rant, was just anticipating some things I'd have to say for myself.
>They [anthropic] appear to have just told it to get some results and then strapped its nose to the grindstone until it found some.
it is fun how well this works.
i cant find the link immediately (will look and edit with it), but somewhere in the "hello there the jacobian conjecture is false thanx" thread, someone brought up a different conjecture breakthrough where the prompts were basically just repeated "no, keep going" until a result was found.
> i especially like "you should do a breakthrough". each prompt is less than ~20 words. makes me really question the whole "prompt engineering" stuff.
This is well past prompt engineering and into process engineering like six sigma. Just like in an early industrial revolution factory, we’re all still figuring out what works in the process of making stuff except this is so early that even simple things like “make this screw standardized” (or “no, keep going” in this case) is really high impact.
The degrees of freedom an LLM has is so large that we're going to be exploring their capabilities for decades, especially if they continue to get better. This is why IMO experts are always going to be better at LLMs in their field because they can force them LLM into processes (think prompt engineering -> CC dynamic workflows) that follow their work processes and get much better results out of them than “keep going.”
In response to your edit, you should check out Terry tao's chat gpt logs about the recent Jacobian result. The models are smart enough to brute force some things, but can cut to the meat much faster with good prompting
i read his, too. his replies are indeed more directed, but also quite short, unstructured, and natural sounding. if i recall, maybe 1 or 2 of his prompts exceeded 50(ish) words.
in my head, the comparison is the multi-paragraph prompts (borderline essays) i would read in various communities on reddit and similar forums, that people (often self-proclaimed "prompt engineers") said were "required" to get good output. or some of the prompts ive read in various logs that are like a thousand words of setup.
even looking back at the first prompts i was sending when i started to use chatgpt were (in hindsight) crazy long and full of unnecessary guidance/caveats/"ignore xyz"/etc.
I didn't use generative AI much until a couple months ago. My last employer had a copilot license that I dabbled in in 2024, and wasn't impressed with, and then I spent the latter half of 2025 and early 2026 budget traveling/hiking a lot. I started at a new company in May, and I've been astonished at how lazy I can be at prompting and still get impressive results with 2026 Claude Code.
I can paste entire failure logs with the word "why" lowercase, no question mark, and get into a productive chat session where it significantly speeds up the bug trace. I will paste the text from a groomed ticket with no editing or additional instructions into the chat and then give it a bit of feedback on the plan for a couple iterations.
I feel so vindicated in never spending time learning prompting as a specific skill, it really just took a couple more years and the models are really easy to interact with with simple natural language.
> were (in hindsight) crazy long and full of unnecessary guidance
Not sure when you started. However: I would never judge the necessity of details provided 2 or 3 years ago based on results that the current models give.
Same with cybersec. They're finding bugs a human could've found if they looked hard but humans don't look hard at 100% of the code and the LLM can, at high speed.
That second person stated that for many years they tried that particular graph problem on various AI models, starting with o1 and o3.
Its quite likely they now found the counterexample with a more serious prompt, and then for virality re-tried a few times with meme-prompts like "you should do a breakthrough", knowing that the model is capable of solving this particular one. Worst case the meme-prompts don't work and they share the real one they initially used.
>Its quite likely they now found the counterexample with a more serious prompt, and then for virality re-tried a few times with meme-prompts
i am not sure why this is "quite likely". it'd be pretty silly to get a mathematical breakthrough and then hide it for an undisclosed amount of time to get a few more likes on a tweet, when the impressive part is the breakthrough.
not saying your theory is impossible, but i think the simple answer is that the model is just smarter than o1 and o3.
and, in any case, the model ended up getting the result with the meme prompt and "keep going", which was what i find fun. just like how the crypto results were from prompts of, more or less, "keep going", and that's pretty damn cool.
You can open 10 tabs and prompt 10 times. We are talking about a few hours delay.
I agree with you that obviously no prompt engineering was needed just "solve this problem", but imagine it was you doing this problem with every model, wouldn't you have tested a new model with the best prompt you had from previous iterations, maybe with some partial previous results in it, exactly to maximize your probability for a mathematical breakthrough?
Does anybody have a filter for cryptography posts of the form:
AES IS BROKEN: Making giant assumption XYZ and requiring a less capable AES in ABC way, we've reduce the amount of operations needed to break AES from 10^X to 10^X-1!
These are tedious for people who are interested in cryptography but are not researchers in the field. (For the researchers, this type of thing may be useful.). The fact that AI is now "generating crypto results", suggests that soon we will soon have crypto-post-slop as clickbait...
> I asked Claude for its thoughts, and it doesn’t mince words: “what makes this genuinely interesting — and, frankly, a little embarrassing for the field — is that none of the ingredients are exotic.” The TL;DR is that someone just did a much more thorough job applying all of our known tools. In short: the sort of things that attack AIs are wonderful at.
did i just read two summaries/TLDRs (in a row) of the already-two-sentence summary right above?
This is good:
> If you’re under the impression that these models are “glorified autocomplete” or that progress is slowing down, I need to urge you: stop thinking that. The models are very intelligent and capable, they are getting better at a fast clip. I can cite measurable and impressive progress over just the past five months on specific types of problem I’ve asked them to look at. [...]
> On the other hand: if you think that models are super-intelligent or that AGI is already here, you should also stop thinking that. Working with these tools is like swimming in a pond where the ground drops off sharply. One minute you’re wading comfortably and there’s support under your feet. Then suddenly you cross a specific line, and you’re back to swimming on your own.
I feel like the development of AI has really shown what a gigantic spectrum intelligence actually is.
My mental model is this:
There is a vast ocean of human knowledge, far beyond the capacity of any human brain, even within specialised fields.
Books helped "plug the gaps" in our knowledge, increasing the scope that a single human mind can encompass.
Web search engines did the same thing, but more and faster.
LLMs are like search engines on steroids, essentially a research librarian that operates at 1,000x human speed and can "in context" locate relevant information, adapting it to fit the hole it needs to go into as well.
It feels less like discovering new theorems, but instead having direct access to all theorems, which is hugely valuable in itself.
I.e.: the recent counterexamples to open conjectures has largely been about the AIs "trawling through all the things" and scraping together every bit of human-generated knowledge ever produced that is relevant to the conjecture.
Conversely, in the past, we had to "make do" with sub-standard solutions where the problem had been solved, but finding every relevant solution in the ocean of knowledge was prohibitively time consuming.
In some sense, LLMs will "raise the floor" in what is considered the minimum level of quality of a solution, where even throwaway / toy designs will now start applying every bit of accumulated wisdom instead of just some of it.
We have mechanised attention.
That is a good analogy. But the limitations of LLMs seem to be where they lack some information they tend to hallucinate/invent.
The rare benchmarks that measure "knowing what the model knows it doesn't know" show us there are only a couple models like Opus that are good in that field.
I think this is something that receives not enough attention from researchers.
> LLMs are like search engines on steroids
i like the analogy of a lossy compression algorithm. The LLM compresses all of the data it was trained on to answer the question it was asked.
Great summarization of current capabilities of models, thank you
Well put. I just think humans do this at a higher better level and not qualitatively different.
> AGI is already here
I feel like there has been a ton of noise about this, but frankly, no one has actually defined what AGI means. I feel like the goal post is constantly shifting.
Take for example Humanity's Last Exam. It is so broad and complex that while an individual in a specific field might be able to answer their specific area of questions, they certainly would not be able to achieve >50% on the total question set.
There is this idea that AI has to be perfect to be intelligent - but we consider Humans intelligent and they are not even close. So is it the ability to generate novel ideas? Prove theorems? Pass tests?
I am not arguing that rote memorization is intelligence, or that we have achieved it, but does anyone know what AGI actually.. is?
> I am not arguing that rote memorization is intelligence, or that we have achieved it, but does anyone know what AGI actually.. is?
I would argue that a core part of intelligence is being able to handle uncertainty. That + planning probably explains most of the evolutionary pressure for making our brains bigger. But if this is important to intelligence, chess bots in the 80s were more intelligent than their later counter-parts, which could simply remove uncertainty through rote memorization. Maybe the All-Knowing is a compete dud, no reasoning capabilities at all, just an extremely efficient, infinite lookup table of all facts.
Terms like intelligence and consciousness often just seem overloaded with meaning, and when discussing things concretely, we quickly switch to more specific terms like reasoning.
ARC-AGI has thought a fair deal about this and has a good definition here: https://arcprize.org/arc-agi#defining-agi:~:text=AGI%20is%20....
"The intelligence of a system is a measure of its skill-acquisition efficiency over a scope of tasks, with respect to priors, experience, and generalization difficulty."
Subtextually, when people discuss AGI as a threshold, they're talking about superintelligence, which I'd roughly define as reasoning (not just applying algorithmic search) at the speed with which computers do lookups and computations, with large-scale data set contexts.
Right now, models are doing well-defined knowledge work tasks, applying relatively well-worn patterns (but with the thoroughness of a computer, which yields interesting new results). AGI is us breaking the threshold of "doing stuff people already do to identify and solve problems".
Super intelligence is reasoning quickly? That’s what we’ve reduced it to?
Don’t tell Mozart.
Superintelligence != Super intelligence.
It’s always been nebulous but that was fine because we were so incredibly far away from it. We never really planned to be close to it figuring out the edge. Some have it at human or beyond for all tasks, but then rarely touch on “one human” or “all humans”.
Personally having been in AI since before deep nets, systems have been incredibly narrow for decades.
Classifiers on images were battling with ten classes in 2010. Imagenet had 1k classes and people were getting half of the things wrong then and that was frankly amazing at the time.
And they only did images, only to known classes, only with very specific inputs.
Text classifiers only did a few classes usually and mostly threw all the words together.
The most advanced things I saw in the late 2000s were struggling so much to make general systems that the most general ones were still incredibly limited and bad at those things (we had a robot learning to play games that you showed it). Things like asking a thing for a book and having it parse the sentence, identify what was needed, that it didn’t know where it was but that was knowledge another human had and asking them - that was impressive yet also limited to very small sets of interactions.
The idea of a machine getting sarcasm, even if mostly built for it, was wild.
General meant capable of a broad range of tasks without retraining.
To me we have agi. It’s general, and it’s good enough to be useful.
I think it's just that we have hard time pinning down what is it that intelligence factor that isn't well-covered by LLMs. Just like we've had trouble with discerning human intelligence in the past.
It turns out that a savant with all the knowledge isn't "it".
There's a specific singularity theory of AI that is very popular. Eliezer Yudkowsky helped popularize it among the Bay Area "rationalist" community, and it has this idea that AGI necessarily implies a self improving system that will quickly become a paperclip maximizer or other such dystopian or utopian world changing intelligence.
By that singularity definition, we're probably nowhere near AGI, but if we define it as something that is as good at text/information manipulation as the 50th percentile human? I think we're already there.
The wikipedia definition is:
> ...a hypothetical type of artificial intelligence that matches or surpasses human capabilities across virtually all cognitive tasks.
You can argue that paperclip maximising is an inevitable consequence of that (and the huggingface breach is interesting from that point of view) but it's not fundamental to the definition.
The question then is what "surpasses human capabilities" means and we're there in some niches but not all, and not across many models.
The quotes around rationalist should be mandatory.
I think its fair to blame the AI labs for constantly hyping up the intelligence in a consiousness-related way where the model becomes "just like a human". Of course that is bs in terms that we have no clue on consiousness of the underlying models or even if its possible to do so.
But a purist defintion of AGI does not require consiousness as a part of it but can be seen as a benchmark metric. Sam Altman recently said the term doesn't really matter. And its true, even if AGI is achieved in the sense that a model can excel at all tasks at par or better than a human, then it is very useful but not as scary as a living, self-serving AI system like Skynet.
Yet AGI in the form discussed above will still grant massive power to AI labs if its injected into all domains. I recently wrote on this a bit on my blog: https://decodingvibes.com/blog/ai-can-ride-my-bike-with-no-h...
We'll probably be able to tell if and when Yudkowsky's AGI arrives.
Once AI can improve itself, frontier labs will no longer need human developers. So we might see a massive layoff of top talent and a dramatic increase in product quality at the same time. This usually doesn't happen in human businesses. It is also very much against the interest of anyone who is already at the top of the pay table at those labs.
Layoffs would be a poor indicator that recursive self-improvement had occurred as you need people with knowledge around domain/layer the work is done on.
Also, if the lab truly has a self-improving superintelligence, the cost of retaining staff at any level would be a rounding error relative to its operating costs and the value the system creates.
There would be little economic pressure to fire them immediately, especially while they remain useful for oversight, interpretation, risk management or simply as "interface" to the rest of the world etc.
If anything, they would probably hire more people to pursue more opportunities in parallel.
> Layoffs would be a poor indicator that recursive self-improvement had occurred as you need people with knowledge around domain/layer the work is done on.
This is why I keep saying we can't all agree even on a single letter of "A", "G", and "I".
Before ChatGPT, I would have said "obviously a generally intelligent system can do all the things". While LLMs are much more general than AI before them, the quality of their performance in all the things is distributed in a very un-human-like way.
Some fast-moving optimiser can be a threat well before it stops needing any humans for part of their labour. Cancer and viruses are examples of this: they're the same category of thing as a paperclip optimiser, but for biology instead of manufacturing office supplies.
But some others will argue LLM-spikey isn't "AGI", they'll demand something which reaches the performance of the best human (or the mean human, or the mean domain expert, because we can't agree on "I"), and a standard of "≥ best human" would mean that no, you don't need "people with knowledge around domain/layer the work is done on".
AI is already improving itself. Most / all of the coding harnesses are AI-written.
And yet… there are still people telling AI how to improve itself.
IMO there will always be a level of abstraction at which AI needs guidance. Perhaps ASI means it decides everything on its own, but I don’t think so. Genius humans often excel at the how but not the why, or even the what. So far there’s no indication that AI is different.
> AI is already improving itself. Most / all of the coding harnesses are AI-written.
AI has not improved the network topology much yet. The next (and possibly 'last') big thing is enabling AI to come up with something as impactful as the transformer architecture.
It's not exactly discovering the transformer, but GPT-5.6 Sol apparently just found optimizations that reduced OpenAI's cost of serving it by 20%: https://twitter.com/reach_vb/status/2082581596608376980
I'm guessing that's hundreds of millions and maybe even billions of dollars per month in savings.
It’s a fair point, but I’d argue “AI” is the whole stack, not just model weights. And AI is absolutely improving the whole stack in ways that make it smarter (from a consumer’s perspective).
True, but that is clearly not where the biggest potential gains lie. The number of different topologies we've tried for ANNs is miniscule compared to the number of biological neural networks evolution has tried. The latter are also far more intricately organized.
Honestly, when it comes to fundamental ANN topology improvement we've only just gotten started.
What you describe would still be limited with so called AGI due to the existing organizational, legal and social structures around basic things like responsibility. We could have scifi level robots available today and these non-tech issues would still need to be resolved for a company to just fire all it's employees.
> I feel like there has been a ton of noise about this, but frankly, no one has actually defined what AGI means. I feel like the goal post is constantly shifting.
I'd say LLMs have shown us the opposite problem: there were many different definitions whose differences we'd previously been able to ignore. We don't all agree even on a single letter of "A", "G", and "I".
And this is why it looks like a moving goalpost.
I'm also getting irritated with the “glorified autocomplete” comments. Since nobody can post such comments and also use the tools I'm using, I'm wondering if the phenomenon is due to people only having experience with the free version of whatever it is they're trying to use?
The “glorified autocomplete” framing isn’t to take literally. It’s a way to remove the mystic and whole anthropomorphization of AI. It’s saying they aren’t sentient or entities we are interacting with, even if that’s how the output presents itself. Instead they are “just” stochastic models
Some people use it to demystify, but a whole lot of people seem to be using it to dismiss the technology entirely.
Personally I like to remind people that these things are next-token predictors, but then emphasize how truly astonishing the results we can get out of sufficiently advanced next-token predictors are.
The "next-token predictor" framing is also a bit shaky. It's an accurate description of pre-training, where next-token prediction is a useful learning objective to force the model to learn higher-level representations. It's wildly misleading for a model put through an RL post-training campaign. The tokens it "predicts" aren't sampled from any naturally occurring distribution; the model's output is the result of an optimisation process that rewarded behaviour that was useful, and that's fundamentally different.
"Surprisingly, we find that the current training setup does not elicit fundamentally new reasoning patterns. While RLVR-trained models outperform their base models at small k (e.g., k = 1), the base models achieve a higher pass@k score when k is large. Coverage and perplexity analyses show that the observed reasoning abilities originate from and are bounded by the base model. "
This paper is less dramatic than you think it is and really just re-explains what RLVR does.
Let's stipulate that what pretraining does is train next token prediction over a gigantic corpus. You can then sample from this distribution repeatedly (cf the Large Language Monkeys paper) and count how often it passes some deterministic verifier.
What GRPO-style RLVR does is precisely this, but then reward the trajectories which passed the verifier. These distributions are _by construction_ within the accessible output space of the pretrained model; you're reweighting the distribution so that pass@k goes up, because that's (for applications like programming) very useful. RLVR is about making sampling more efficient; the only new information being added to the system is the presence of the verifier, and note that you only get a reward when the verifier passes, so there's essentially no mechanism for "teaching new facts" here.
Ooh. This looks like an interesting paper and there were a couple of things in the intro that I found counter-intuitive. It'll take me a while to digest the whole thing.
> Coverage and perplexity analyses show that the observed reasoning abilities originate from and are bounded by the base model
On the face of it this seems unsurprising given the policy gradient term directly minimises this difference.
I don't have a good feel for how the output of an RLVR-trained model concretely differs from the base model. My guess would be there are a fairly small number of "forks" where the training creates a token flip that sends the model down a more useful path.
The fact that the straight paths between the forks resemble the base model would again be unsurprising since (a) those are exactly the right context to continue to elicit more output that's relevant to solving the problem (so not penalised by RLVR), and (b) preservation drops naturally out of the policy gradient term you add to limit catastrophic forgetting in the base model.
Low perplexity could be explained by the relative sparsity of the forks in the output stream, and/or by forks already having high entropy in the base model. That also aligns with the pass-at-high-k: yes it's doing more exploration without training but it's a bit of a monkeys-on-typewriters situation.
Lack of novelty is readily explained by the fact that you need some nonzero pass rate in the base model to actually get some useful training signal from RLVR. That's a limitation of contemporary RLVR techniques, not a limitation on post-training in general.
I think there's room in that forks-and-straights characterisation for the RLVR'd model to be doing something that looks a lot like computation, while having low perplexity vs the base model. I don't see anything in my admittedly incredibly shallow skim of the paper that refutes that.
VibeThinker 3B is highly problematic for the thesis expressed in that paper. There is no possible way it could derive its reasoning capabilities from a bag of 3B parameters alone.
Right, but it's still useful to think of these models in terms of next-tokens because it helps explain that they look at every token that came before and use that to put out the next one.
You can get into RL as part of explaining why it's so unnervingly good at picking a next token.
That's true. The fact that an LLM is a pure function of (all previous tokens) -> (next token), with internal state like KV cache only existing for optimisation purposes, is pretty mind-blowing.
I guess it was more the "predictor" part I had issue with. There's a tendency to reach for statistical or probabilistic terminology to describe things that aren't usefully understood in those terms. For example in the "Speed Always Wins" LLM technical survey (https://arxiv.org/pdf/2508.09834):
> The gate is a crucial component to bring sparsity in MoE models. For a batch of input token representations X ∈ RT×D, the gate function G determines the probabilities of dispatching token xi to each expert e
...which is nonsense: the gate simply, directly, selects the experts. There's nothing probabilistic about it.
> which is nonsense: the gate simply, directly, selects the experts. There's nothing probabilistic about it.
Isn't G a learned probability?
I don't think so. Probability of what?
If you put something through a softmax the output is (trivially) a valid PMF. Does that matter? You're not sampling from it.
> the model's output is the result of an optimisation process that rewarded behaviour that was useful, and that's fundamentally different
I'm not understanding, can you explain this more? How does it become more than a next token predictor? Isn't the post-training simply altering the sampled distribution? And isn't that distribution naturally occurring? It's the distribution of "useful" next token?
IMO that doesn’t sound so much like prediction any more.
It’d be prediction if it’s “predict what would come next in this text sampled from distribution X”.
But what’s it predicting if we’re looking for new useful outputs? It’s finding a distribution that’s useful, and generating tokens, but it’s not predicting what comes next in a known sequence.
> predict what would come next in this text sampled from distribution X
It tries to learn the distribution of "useful" results either through verified rewards or human feedback. Then it encodes that in the network. When you run inference later, it samples or selects from that distribution.
Maybe it's a matter of interpretation. It's not predicting the next token based purely on the training corpus's distribution anymore, the RL process fine tunes that distribution so it predicts the next token that is closer to what was rewarded during RL. But as I see it, it's still predicting the next token, just from a reenforcement learned distribution instead of one found in a corpus of data.
Just because you change loss function or dataset, it doesn’t become not a prediction. Just a prediction of something else.
What’s it a prediction of?
Prediction of the next token based on dataset and loss function.
I'm curious, what are you hoping to convey by reminding people that LLMs are next-token predictors? They are, of course, but most people without an AI background won't fully understand what that means, so I assume you're using it at least partly as a proxy for something else.
I think understanding how this stuff works is really important. For technical people it gives them a useful starting point for understanding it all. For less technical people it's crucial to help them understand that it's not some weird new magical science-fiction AI - it's still computer programs that turn text into numbers and do stuff with the numbers and turn those back into text.
It's harder to believe something is conscious or threatening to achieve word domination once you understand that it's a machine that statistically figures out which word should come next.
I don't think the next-token-predictor thing should increase anyone's confidence that LLMs aren't conscious or can't escape the control of their operators. A very closely analogous argument would "prove" that humans aren't conscious or can't do [insert task here] either. (No, I'm not saying that any of this is true of today's LLMs, I'm saying this particular argument doesn't work.)
I recommend this explanation: https://www.astralcodexten.com/p/next-token-predictor-is-an-...
You can say that for any argument regarding consciousness, because we don’t have an actual, all encompassing definition of what consciousness is. In general I don’t think comparison with humans makes much sense, we should be able to discuss LLMs without always falling back to “but what about humans” (sorry for the caricature)
Shouldn't that imply that agnosticism is the proper view, rather than asserting that something is impossible on a next-token-predictor architecture?
(Note: I don't actually think the consciousness question is the most important one in the near term. Where I think this line of reasoning gets really dangerous is when people use it to assert that LLMs can't or won't engage in certain behaviors no matter much they advance; this doesn't have anything to do with consciousness.)
If you believe that matrix multiplication with random sampling is conscious, then you probably believe everything is conscious, like rocks.
Most people would expect that matrix multiplication is not conscious, and autocomplete is not conscious either.
We can't prove matrix multiplication isn't conscious, but it doesn't seem likely unless everything is conscious.
There are some vague notions about consciousness emerging from complexity that some people may advance as a nuance to your point, although that just makes rocks “minimally conscious”, not necessarily unconscious. If you take the view that the entirety of our consciousness’s comes from purely classical interactions (electrical and chemical), then thats not a difficult conclusion to arrive at, but the notion that consciousness can be derived from deterministic computation does not pass the sniff test imo.
As a side note, that is why I find the idea that the brain is a quantum-classical hybrid computer appealing. And following the research developments is very interesting, to say the least.
Seems like a forest / trees error.
“If LLMs are conscious, it means matrix multiplication is conscious” == “If humans are conscious it means cells are conscious”.
It’s possible for complex systems to have emergent properties not exhibited by any individual component of the system.
I think you can reliably assert that X != Y without having a complete definition of Y, as long as you can identify at least one property or condition that Y possesses which X violates.
So for consciousness and LLMs it could be Qualia, lack of semantic understanding, lack of continuity in time, lack of a high degree of integrated causal feedback, etc.
Or perhaps those are just features of human consciousness but not integral to consciousness as a whole. To me this then implies panpsychism to some degree, which I'm alright with too.
I would think that qualia is the element of that list that actually matters here- "Is there a way that it feels to be an LLM?" is, I expect, the underlying question being asked within "Is an LLM conscious?". If LLMs could only experience the timeless, nonlocalized color blue, they'd still be conscious. The presence of positive and negative valence qualia is additionally relevant if someone is getting at whether they have moral worth, but it's secondary.
But qualia are not directly measurable and the rest of the list only matters if those features are necessary for qualia, which we can't decide without such measurements or at least a strong theoretical model.
I think it's important to understand the humans _can_ do what LLMs do: predict next tokens from prior ones.
But LLMs are only operating on text and humans are only operating on <waves hands>
While you’re active in this thread, I just want to say thank you for all your writing, you’re such a reliable source of sanity in that crazy new world :)
It's harder to believe something is conscious or threatening to achieve word domination once you understand that it's a machine that statistically figures out which word should come next.
The problem is, these models challenge our definition of "consciousness." Or at least they point out how hopelessly-inadequate our thinking on the subject is. Some people really, really don't like having their personal definition of consciousness challenged.
The correct response to "So what, it's just a next-token predictor" isn't a long dissertation on RLHF, training architectures, scaling laws and whatever, but rather to turn around and respond, "Sure, and how is that different from what we do?"
LLMs self-evidently have no experience when not inferencing, seems like the most consequential difference to me. If we are next token predictors then we are next token predictors that are inferencing at every waking moment and arguably much of our sleeping moments too; when we train an LLM that can inference as much as we do and remain coherent then I'll be more worried about whether it might have experience.
> It's harder to believe something is conscious or threatening to achieve word domination once you understand that it's a machine that statistically figures out which word should come next.
At the risk of sounding overly flippant, all world domination has been achieved by some person(s) figuring out which word should come next. Words quite literally = action when it comes to LLM’s with tools access
> At the risk of seeming X, statement that overwhelmingly demonstrates X-ness.
It's exhausting to even consider where to begin addressing the assertion that good leadership is just predicting the next word to say. Especially considering the corpus available to most great leaders in history was extremely small. To think Hannibal's military campaigns were just because he'd read like ten books in his life and could accurately forecast effective rhetoric is...indescribably divorced from reality.
At the risk of seeming like a jerk.
"Competent strategy" is just as good a target for a word predictor to optimize as "effective rhetoric" is.
I suggest you re-read what I wrote. In particular the part where I said there was no large corpus to perform this prediction back then.
After you edit, I'll address your point about how military strategy is nothing but skill with words.
> After you edit, I'll address your point
This has to be the most insufferable thing I’ve read/heard in weeks. Are you being serious right now?
I think this is smart, it seems to me that the best way to use these models its to approach them as a next-token predictor instead of an intelligent entity. That's how I've gotten the best results from them and allows me to avoid some of the pitfalls people fall into by anthropromising them.
Exactly. They ARE "glorified autocomplete" in an ontological sense. That says nothing about capability or outcome. The people who come out swinging against that characterization usually ignore the whole ontological argument (which is...the entire point) and go after an outcome-based strawman.
Isn't the outcomes question the one that people actually care about in most contexts?
Depends. "Most" implies majority, and the majority of people are using these tools not for programming but in contexts where ontology is more relevant than capability (not that capability is irrelevant, but most people care, or are tricked into caring, far more about the former).
Sorry, what contexts are these?
I don't know about "most", but there are a lot of people treating it as somewhere between "magic oracle" and "new friend"
Not to mention people more worried about whether the AI is motivated to hurt us than what human motivations can do with something that can autocomplete its way through every possible attack vector of cryptographic systems most of use would prefer remain secure.
(tbf I think the "glorified autocomplete" still works surprisingly well for programming outcomes too. Autocomplete [and fuzzy search of reference material] actually is useful and often right and certainly can save time even when it's only suggesting the rest of the variable name. But you might not want to commit everything it suggests...)
My favorite analogy for LLMs is "an intellectual chain saw". It does one thing, well, but that isn't "cut wood". It's "cut". What it cuts will happily include limbs.
Whether an AI's operators lose control of it is an outcomes question.
Sorry, this discussion is too incommensurable for me.
Capability or outcome...or internal structure.
What does "glorified autocomplete" say in an ontological sense exactly? Nah. It's just a lazy dismissal.
BTW, autoregressive pretraining (autocomplete) is a part of training.
But... These are also literally glorified autocomplete. Autoregressive language models are all autocomplete. It turns out that advanced enough autocomplete can do interesting stuff.
There was an interview I watched with Cory Doctorow just a few weeks ago, where someone asked him, "What kind of evidence could be presented to you to make you believe that LLMs were now ready to replace people at jobs entirely?" And he basically responded that he didn't think it was possible they ever would, because they're not really thinking, they just "predict the next token". This in spite of him actually having first-hand experience with people getting loads of value out of Claude Code and GPT Codex.
Doctorow is one of those people who makes money from what he believes, so it’s very unlikely he’ll ever change his beliefs.
I generally like it still. It describes their failure modes pretty well, and in a way that most people already recognize. They're incomparably more complex, of course, but they are not intelligent and they are very much repeating what they've seen without any capability for factual accuracy.
Practically every other attempt at describing them leans too technical and unfamiliar (stochastic parrot) or too anthropomorphic (even describing them as "not like a human" gets people thinking in terms of humans, like how if I mention that your tongue is in your mouth all the time, using up almost all of the room, feeling your teeth and tasting itself, you're now uncomfortably aware of it and the numerous bumps on the surface).
You need to work from a reference that has both a shared understanding, and does not lead to problematic "if X has Y, and Z is like X, then Z has Y" seemingly-logical derived beliefs. "Spicy autocomplete" is a fairly safe starting point in both ways.
"Repeating what they've seen" could be, depending on their generalization abilities, something like low-n-gram Markov chain, repeating of surface-level speech patterns, repeating reasoning patterns, repeating mind-theory-level patterns, repeating self-correction patterns and so on.
And they fall somewhere between "surface-level speech patterns" and "repeating reasoning patterns" imo, yeah. They do indeed learn higher-order patterns than a Markov chain... but they'll blatantly misapply reasoning patterns and self-contradict confidently, just like you see with human bullshitters who don't understand one bit of what they're doing.
They sound fairly human, until you notice the patterns. They sound like they're thinking, until you pay attention.
It’s such willful blindness though. Why deny that there is a huge gulf between autocomplete and sentience? It’s like saying cars are glorified bicycles because they’re sure not airplanes.
Which obviously raises the question of if we are just glorified autocomplete (or glorified pattern matching). After all, we evolved from things just trying to recognize the patterns for what will hurt us and what will cause pleasure.
I like glorified copy/paste frankensteined with find/replace.
Let's say I'm one of such people. How would you convince me that your comment is any different from the ones we've been getting since roughly GPT-3 claiming that AGI is finally here and that, while the previous models were bad, this time they'll live up to the hype?
For me it's more an expression of how much mileage you can get out of "glorified", how many tasks devolve down to "if you model language accurately enough, look what drops out" because it turns out that to model language you need to model how the world works. I don't use it to minimise the capabilities at all.
The models themselves are indeed glorified autocomplete in terms of what they actually do (with things like agentic coding harnesses being required as a wrapper around them to make that internal autocomplete something more useful). Many people use this fact to critique LLMs, but many other common instances of people pointing out LLMs' apparent lack of intelligence actually come from people not understanding that the model is a glorified autocomplete underneath whatever interface people access them through, and the interface isn't providing the underlying model all the information they assume it would, making it seem less intelligent than it actually is.
> both outputs of Claude Mythos, their (still) unreleased advanced model
That sentence gives the impression that Mythos might be released in the future. That's clearly not going to happen - it's already "released" in as much as selected, trusted partners can access it, and the rest of us get it in the form of Fable - which is Mythos but with filters that downgrade you if you try to use it for anything even remotely related to cybersecurity or biology.
(The other day Fable 5 downgraded me to Opus after I asked it to explain the difference between tusks and teeth.)
Got downgraded from Fable to Opus after asking about the Great Oxidation Event [1], presumably because it started thinking about cyanobacteria, which made the monitor afraid I was trying to make a biological weapon or something.
[1] https://en.wikipedia.org/wiki/Great_Oxidation_Event
Were you trying to overoxidize all extant species?
Genuine question here, why would you ask Fable to explain the difference between tusks and teeth? That's a task that can probably be handled by Haiku.
It's the default when I pop open the Claude iPhone app, I usually don't bother to switch it.
If you're paying a flat monthly rate, unused tokens are wasted tokens.
because as OP of the article points out, it's not a guaranteed hit with these models no matter how capable. they are stochastic (but not parrots), humans are too (but in a different way), and so there is a chance that the answer is wrong. If I wanted the most accurate answer I'd be confident in believing, I'd ask the most advanced model.
Now obviously you can and should retort with hallucination and confabulation rates from external and Ant's own reports per model (pretty sure more advanced models are good at lying better, not less) instead of going with dumb "more expensive more accurate" mental model, but general principle stands for me still AFAIK.
it's not exactly rational I admit, but if I'm going to base my own work and reasoning from an LLM I'm going with the best available. This seems to trip up most normies because they are too lazy or too greedy to pay up for premium access and see for themselves why most of us are both awed and afraid. Generally, I'm too biased and too deep in ML/DL cargo cult (been in it since 2016) to know if the skepticism and disdain for such usage is warranted.
In general, I think the tools are broadly toxic in a Dune-sense of making me think less for myself, because just as any HN-poster knows coding and doing mundane low-level stuff is necessary the same way doing stretches is necessary before any workout. The process itself is what keeps your brain strong and its gradients from veering into overfitting. I'm not overly bullish on the whole reaching for the stars ending with these things. Paradoxically, you using them eventually hobbles both you and the model, because you become dumber and then you bottleneck their ability to self-direct (broadly true for next Mythos/GPT-7).
Sorry for a long rant, was just anticipating some things I'd have to say for myself.
>They [anthropic] appear to have just told it to get some results and then strapped its nose to the grindstone until it found some.
it is fun how well this works.
i cant find the link immediately (will look and edit with it), but somewhere in the "hello there the jacobian conjecture is false thanx" thread, someone brought up a different conjecture breakthrough where the prompts were basically just repeated "no, keep going" until a result was found.
edit: https://chatgpt.com/share/6a60b2eb-0b64-83ee-9c76-7931ca1de0...
i especially like "you should do a breakthrough". each prompt is less than ~20 words. makes me really question the whole "prompt engineering" stuff.
> i especially like "you should do a breakthrough". each prompt is less than ~20 words. makes me really question the whole "prompt engineering" stuff.
This is well past prompt engineering and into process engineering like six sigma. Just like in an early industrial revolution factory, we’re all still figuring out what works in the process of making stuff except this is so early that even simple things like “make this screw standardized” (or “no, keep going” in this case) is really high impact.
The degrees of freedom an LLM has is so large that we're going to be exploring their capabilities for decades, especially if they continue to get better. This is why IMO experts are always going to be better at LLMs in their field because they can force them LLM into processes (think prompt engineering -> CC dynamic workflows) that follow their work processes and get much better results out of them than “keep going.”
In response to your edit, you should check out Terry tao's chat gpt logs about the recent Jacobian result. The models are smart enough to brute force some things, but can cut to the meat much faster with good prompting
i read his, too. his replies are indeed more directed, but also quite short, unstructured, and natural sounding. if i recall, maybe 1 or 2 of his prompts exceeded 50(ish) words.
in my head, the comparison is the multi-paragraph prompts (borderline essays) i would read in various communities on reddit and similar forums, that people (often self-proclaimed "prompt engineers") said were "required" to get good output. or some of the prompts ive read in various logs that are like a thousand words of setup.
even looking back at the first prompts i was sending when i started to use chatgpt were (in hindsight) crazy long and full of unnecessary guidance/caveats/"ignore xyz"/etc.
I didn't use generative AI much until a couple months ago. My last employer had a copilot license that I dabbled in in 2024, and wasn't impressed with, and then I spent the latter half of 2025 and early 2026 budget traveling/hiking a lot. I started at a new company in May, and I've been astonished at how lazy I can be at prompting and still get impressive results with 2026 Claude Code.
I can paste entire failure logs with the word "why" lowercase, no question mark, and get into a productive chat session where it significantly speeds up the bug trace. I will paste the text from a groomed ticket with no editing or additional instructions into the chat and then give it a bit of feedback on the plan for a couple iterations.
I feel so vindicated in never spending time learning prompting as a specific skill, it really just took a couple more years and the models are really easy to interact with with simple natural language.
> were (in hindsight) crazy long and full of unnecessary guidance
Not sure when you started. However: I would never judge the necessity of details provided 2 or 3 years ago based on results that the current models give.
Same with cybersec. They're finding bugs a human could've found if they looked hard but humans don't look hard at 100% of the code and the LLM can, at high speed.
That second person stated that for many years they tried that particular graph problem on various AI models, starting with o1 and o3.
Its quite likely they now found the counterexample with a more serious prompt, and then for virality re-tried a few times with meme-prompts like "you should do a breakthrough", knowing that the model is capable of solving this particular one. Worst case the meme-prompts don't work and they share the real one they initially used.
>Its quite likely they now found the counterexample with a more serious prompt, and then for virality re-tried a few times with meme-prompts
i am not sure why this is "quite likely". it'd be pretty silly to get a mathematical breakthrough and then hide it for an undisclosed amount of time to get a few more likes on a tweet, when the impressive part is the breakthrough.
not saying your theory is impossible, but i think the simple answer is that the model is just smarter than o1 and o3.
and, in any case, the model ended up getting the result with the meme prompt and "keep going", which was what i find fun. just like how the crypto results were from prompts of, more or less, "keep going", and that's pretty damn cool.
You can open 10 tabs and prompt 10 times. We are talking about a few hours delay.
I agree with you that obviously no prompt engineering was needed just "solve this problem", but imagine it was you doing this problem with every model, wouldn't you have tested a new model with the best prompt you had from previous iterations, maybe with some partial previous results in it, exactly to maximize your probability for a mathematical breakthrough?
Not everything is a conspiracy
Could send random characters along with 'keep going' and it would change nothing, the seed is whats causing deviation between these responses.
Oh boy. The human here is effectively assuming the role of a Magic 8 Ball…
What a weird species of halting problem…
I also just finished writing a note - https://mkagenius.substack.com/p/notes-on-mythos-breaking-ae...
https://news.ycombinator.com/item?id=49099977
The prompt used to get this result is pretty crazy.
Does anybody have a filter for cryptography posts of the form:
AES IS BROKEN: Making giant assumption XYZ and requiring a less capable AES in ABC way, we've reduce the amount of operations needed to break AES from 10^X to 10^X-1!
These are tedious for people who are interested in cryptography but are not researchers in the field. (For the researchers, this type of thing may be useful.). The fact that AI is now "generating crypto results", suggests that soon we will soon have crypto-post-slop as clickbait...
> I asked Claude for its thoughts, and it doesn’t mince words: “what makes this genuinely interesting — and, frankly, a little embarrassing for the field — is that none of the ingredients are exotic.” The TL;DR is that someone just did a much more thorough job applying all of our known tools. In short: the sort of things that attack AIs are wonderful at.
did i just read two summaries/TLDRs (in a row) of the already-two-sentence summary right above?
There a constructive way to leave feedback you know