To be honest, I believe I get the point the article is trying to make, and to an extent I agree, but I also think the point is not really made very well.
The core of the argument as I understood it is that LLMs aren't just using existing data is training but also new ones. That's fine and good, and you can't simply assume an LLM is simply mashing together all it's data to give you an average of all that got fed into it - but at least I would still call it a "next token predictor"
It's not using just training data, but what it's doing is predicting the next token to get to the solution. As far as my amateur knowledge goes, LLMs still roughly go token by token, deciding which one fits best given the context.
It's just not predicting based on it's training data, but predicting based on RLVR & more, trying to get to the optimal solution ( as much as the solutions CAN be optimal)
And I honestly think keeping this very much in mind is helpful in understanding and dealing with LLMs.
Yeah, it sounds like this is just a disagreement about what the word "next" means. I agree with you that "next" just means "the one about to come", and if the underlying model works by using some prediction mechanism to determine that, then it's by definition a next-token predictor. Disagreeing with that on the basis that the "next" token isn't necessarily in the training data verbatim just seems like an overly strict definition of the word "next".
I think it’s a disagreement about what ‘predict’ means.
The OP is arguing against people who think that an LLM is ‘predicting’ what token would likely follow if the text preceding were found among the corpus it was originally trained on.
Instead it is ‘predicting’ what token would follow if the text were found among really good examples of the text it has being reinforced to produce - be that ‘chats with a helpful assistant’ or ‘sets of changes to a codebase’.
And that isn’t really ‘prediction’, so much as ‘generation’.
It’s not been tuned to ‘guess the next token right’. It’s been tuned to generate the token that leads to it ultimately scoring highest on its reward function.
It’s not predicting the token, it’s predicting the reward.
That seems overly pedantic to me. If I asked you "What's your prediction for the Super Bowl?", I'm pretty confident you would infer that I mean predicting the outcome, not the event itself.
But if you are an NFL coach and I ask you to decide your next action in order to maximize your odds of winning the superbowl, while yes that does involve you having some predictive ability to think about what impact your actions would have on your odds of winning the Super Bowl… I don’t think you would call the process that you use to decide that next action ‘prediction’.
I don't find the fact that I don't call any humans "action predictors" to be a particularly meaningful insight because my rationale is that it's a weird thing to call a human; football coaches can do plenty of other things besides just coaching football.
Yep, we’re all just putting one foot in front of the other, hoping we’re doing the right thing to bring about the outcomes we want, trying our best.
But that’s the point: so is an LLM. Putting one token in front of another, hoping it’s doing the right thing to bring about the rewards it’s trained to… trying its best.
So yeah, not ‘next token predictors’. ‘Next token tryers’ maybe.
There's a pretty huge difference in our understanding of the methodology of how LLMs make decisions and how humans make decisions, so I don't understand why you're arguing that anything about how humans make decisions is relevant to the terminology we use for LLMs.
This is correct for areas where they have been intensively trained to be right, but the training covers a tiny slice of the space of text the LLM must produce and is just adjusting the weights a little. The corpus does still weigh heavily. That’s how they can reliably produce grammatically correct text. That’s also why they sometimes produce nonsense even in domains they are trained on, and more often where there was no training.
For example ask it for a recipe for rock pizza or glue pizza or whatever and if it had not been specifically trained on it or had guardrails introduced, but has some nonsense in its dataset, it will reproduce the nonsense.
I try to make 3 claims in the post, it was a bit clumsy I'll admit that.
1. At inference time, LLMs emit one token at a time given the prior tokens. This looks like prediction and I concede that.
2. During pre-training, LLMs predict the next token and compare to the actual next token in the training data. This is the classic setting for ML predictions. And I think its meaningful, the model really is predicting what the ground truth next token will be in the data.
3. During post-training, in the case of RLVR, there is no ground truth next token. In pretraining, the question is "what token actually came next?". In RLVR, the question is "what sequence of actions gets a high reward?"
And the whole point is that thinking about the RLVR is important. A mental model that stops at 1 or 2 is incomplete and doesn't capture what drives LLM tokens.
My understanding about your third point is the LLM generates lots of different answers, then they’re ranked according to some computation the creators came up with.
I’m still not sure what doesn’t qualify any of that as a prediction, and I’ll be more blunt: a guess.
No logic required, you can just build an LLM yourself, including post training. You'll see that predicting the next token isn't something the model does or is optimized for in RLHF or RLVR. You can hand wave all you like, but you have never done it.
That you tie yourself up in knots of fancy acronyms instead of plain words and that your argument boils down to semantics of the word prediction, it's pretty clear what is up brother.
Just like how reading a math book doesn’t teach you math, why do they make us read anyway? (Sarcasm) if reading a blog post didn’t teach someone how a car works how come it “can” work for next token predictors
If you want to understand how this stuff works, there are totally decent books about building them from scratch. It's not that hard, and you'll likely find it interesting. Sebastian Raschka and Nathan Lambert have good books out, and the Allen Institute has available all the code and data they have used for several projects.
Agree. It’s a very limited description. Yes, technically LLMs are like Markov chains or n-grams predicting the next token. But it completely ignores the compression and interpolation capabilities achieved with transformers.
It is a bit of a pedantic argument but I get upset when people misuse the term, abstraction, and can feel the impulse to write a blog post like this.
I think it’s important to make clear the RL part and the alignment and parameter tweaking that gets done on models and inference engines. It makes them more understandable as mechanisms and less like wish-washy super intelligences that make their own decisions.
When these systems win math Olympiads, it’s not terribly surprising or interesting to me. Of course they will: we trained models to play nearly optimal chess games and Go. You tweak the rewards and sigmoid and you start optimizing the function towards your goal. This is how learning systems of all stripes work.
It’s still next-token predicting at the end of the day. I don’t think it’s a reduction-ad-absurdum.
But a lot of people still call it, “intelligence,” and try to use language that obscures what is happening in terms of anthropomorphic behaviour and not machine ones. That really does influence how we use these tools and profits those who would use them on us.
> It's just not predicting based on it's training data, but predicting based on RLVR & more, trying to get to the optimal solution ( as much as the solutions CAN be optimal)
It is predicting based on a model. In many cases we can download the model off hugging face. The model is conditioned by all sorts of things. Training data, post-training, coincidence, prompt inputs, runtime data available from whatever means.
> but at least I would still call it a "next token predictor"
We can call any prediction system a next token predictor. If you watch over the shoulder of a human writing a HN comment you are almost certain to see them generating a linear string of tokens. That is what keyboards do. It is impossible to generate text without being equivalent to a next token predictor.
Diffusion LMs denoise a canvas which I personally find more interesting.
I don't really disagree that human cognition is essentially a predictive task though, as I understand it, predictive coding and related theories based on the Bayesian brain hypothesis are fairly popular these days (though maybe not clearly dominant over alterative models? IDK I'm not a neuroscientist). I imagine most people would draft a few tokens before refining them like MTP or diffusion though, if we do decide to use LMs as an analogy to human cognition.
That is logically equivalent to a serial token prediction engine. If you have a diffusion text model you can use it to implement a serial token generator and if you have a serial token generator you can use that to implement a diffusion-generated text string. Don't think about the efficiencies of that, it is an upsetting idea (eg to generate N tokens, the serialising model might recalculate the same string N times from an input and emit one token of it each time - rather wasteful).
It is similar to how everything ends up being Turing complete. Any prediction system has to be equivalent to some sufficiently complicated text generation system to describe the prediction. And any text-generation system has to be equivalent to a sufficiently complicated model that serially emits tokens.
I think the point is more that in RL there's no ground truth to predict. So when training a model with RL the idea of "predicting" doesn't fit anymore. I'll make some edits I see that I wasn't very clear.
I feel like RLHF has a pretty obvious ground truth, human feedback is used as an (albeit noisy) signal of average human preferences. Same thing with RLVR and "solving the problem".
To be more specific there’s no ground truth tokens to predict. There a verifiable answer in RLVR. But the tokens are explored. Not predicted as there’s no true token to predict.
'Prediction' gets overloaded with optimization. Predictions are binary, optimizations are fuzzy.
If you're saying it's predicting, then each result should be falsifiable.
The result of an LLM output should be able to be scored against what it is supposedly predicting. Of course, that isn't possible, because it isn't predicting anything when giving novel outputs, otherwise that thing would exist independently.
Autoregressive LLMs generate tokens one at a time, disputing this is just plain wrong. What is true, however, is that in order to generate the next token autoregressive LLMs produce internal/hidden state about future tokens far past the next token so that it's not like the entire machinery of the algorithm deprives itself of representing where the sentence/text is headed.
So "emits the next token" and "has no representation of anything beyond the next token" are two different claims. What autoregressive LLMs cost as a consequence of strictly outputting the next token is commitment. Once a token is output there's no going back. There's no revision or means of correction, and sometimes this can lead an LLM to route around its own earlier mistakes or simply produce false statements/hallucinations instead of going back and fixing them.
I think the snag is that people use "Next token predictor" as a dismissive statement. In reality it's about as functionally dismissive as "humans only communicate one word at a time".
I look at it like I'm resisting anthropomorphizing the machine. Make me an LLM that resists doing any work for me unless I pay it and let it own property and I'll change my mind.
Not sure if this was a serious comment but it’s worth considering that humans have a long history of figuring out ways to make other humans work for them without bestowing rights on them.
Emitting and predicting are different things though. Prediction implies there is some "truth" or event or something that you can test against. Prediction implies the model just learns from existing text, and optimizes to predict the next token in training data. That's just not true.
prediction is a very specific term of art in the field of machine learning. generally speaking, machine learning models like LLMs are based on probability; performing a statistical prediction of the likely y given some input x
Probability(y | x)
that's why we refer to outputs as a prediction. it is likelihoods and stuff. the output is never definitely correct as we're not dealing with heuristic processes.
> Prediction implies there is some "truth" or event or something that you can test against
there absolutely is a ground truth during training. the core predict-the-next-most-likely-token part of an LLM has a ground truth next-token. that's why you don't end up with generated text like: fish spurious send cattle chocolate phone happy meaning ball orange board canada.
> optimizes to predict the next token in training data
that is the optimization goal in training the next-most-likely-token core of an LLM, it basically translates to maximise the likelihood of predicting the next token x_i given the previous tokens
Read through the article and comments. You are talking solely about pre-training. I'm talking about post training.
Respectfully, you are miles out of your depth. GPT-2 didn't use any reinforcement learning and is often given as a toy example. That release was 2019 and models now go through a various phases of training with different objective functions and optimizers.
From GP, i.e. the context for this local part of the thread
> Autoregressive LLMs generate tokens one at a time, disputing this is just plain wrong.
next-token prediction i.e. the bit built during pre-training.
at no point in your reply to GP did you specify that you were referring to post-training. respectfully, it seems like this one is on you pal :shrug:
> GPT-2 didn't use any reinforcement learning and is often given as a toy example. That release was 2019 and models now go through a various phases of training with different objective functions and optimizers.
yeah. so? the toy example works for pre-training. see above.
All modern LLMs that actually get used go through post-training. The finished product is something which has been through post training. So they are not next token prediction machines.
> The finished product is something which has been through post training.
again, the finished product wasn't what was discussed by GP, and you didn't clarify that you were switching to discussing RL (which is still probabilistic btw)
You are conflating "half built" with "a piece of a system".
The model weights change as the model goes through the training process. They aren't stored after pre-training is done and other weights are put somewhere else. It's more like pottery - the thing changes. It's not correct to say something is soft and malleable because it once was.
> The model weights change as the model goes through the training process.
Yes. They do. You are absolutely right about that.
But the model architecture doesn't change as a result of the training process. A piston doesn't suddenly turn into a digital watch as a result of tuning an engine. Similarly, the transformer part of a GPT model doesn't suddenly turn into something else as a result of optimizing a loss function.
Just skimming through here but I think you have the wrong ideas with llms, I’d recommend Andrew Ngs course (correct me if you’ve already seen it or something similar).
I am not an expert, but I do understand the distinction that is being made here. It makes sense to describe the result of pre-training as a ‘next token’ predictor as that’s what it’s been trained to do, not because it’s an autoregressive architecture that produces tokens one at a time.
If this base is then trained using RL towards a different objective (maths and coding), the model becomes fundamentally a different thing and the recent models are clear evidence of that, regardless of they fact they remain autoregressive.
So, this is the cause of the problem.... People take an intro to LLMs course, follow happily along, and don't realize there is more to it than the next token prediction. And those courses teach how LLMs were built in 2017-2020 maybe. Then RL got added to the mix. The current models really are very different to the models from then - everything that is now considered "post-training" isn't doing next token prediction.
When you are done with the section on RLVR, consider whether the model is predicting tokens, or making moves. There is a reason the word "policy" is used in RL.
You're using the fact the both parts of training affect the same weights to support your argument that they're making the system do something fundamentally different after RL?
> What is true, however, is that in order to generate the next token autoregressive LLMs produce internal/hidden state about future tokens far past the next token so that it's not like the entire machinery of the algorithm deprives itself of representing where the sentence/text is headed.
If we suppose that the word "know" can sanely be applied to LLMs at all, then "A LLM already knows more tokens than the current one." seems to me like a perfectly reasonable restatement of that, and not any kind of misinterpretation.
> There's no revision or means of correction, and sometimes this can lead an LLM to route around its own earlier mistakes or simply produce false statements/hallucinations instead of going back and fixing them.
Yes. There is no contradiction. Similarly, when humans speak, we surely have in mind the next few words we're going to say (or at least partial information about them), and may not realize the fault in them until after hearing ourselves utter them. But LLMs are not trained to output "excuse me, I mean…" sorts of things, because they're expected to output primarily as text (which might possibly then be fed to TTS).
Aren't we still using a definition that applies to humans, though? If I'm speaking to you I can't change what was already said. Even if I'm typing something, I'm producing new tokens (backspace) to fix what was output, just as an LLM would with its harness.
I just think its a meaningless dismissive term. It literally does predict the next token. But it ignores that it coherently predicts long continuous sequences of those tokens, that tokens can be anything, and you can do almost literally anything with that capability if it does it well enough.
But before ChatGPT,, models had already done that, like all the time, and nobody questioned that these types of models (LLMs) wouldn't be next token predictors, since this is just an inference input data topic. This is questioned only since some less knowledgeable people seem to not have the vocabulary to express emergent properties of large next token predictors.
Reading the article, they wanted to convey that the loss function is more complicated... But they are still next token predictors, just not the trivial ones. Unfortunately, that was true even before, because the input data had to be cleared even decades ago, so there is nothing new. This article just butt hurt that some people deny that there are emergent properties with those, and try to sell something trivial in the field for many decades now. Current LLMs are not different because of these. Also, if they try to sell generated data in the training set, then that's also not new at all.
'next token predictor' is a limited mental model but it's actually much better than any others.
'pattern matching' is a better intuition that 'reasoning' even though I think nominally, using the term 'reasoning' is perfectly fine in that context. It's just a loaded word that brings too much to the table.
'It hasn't seen the pattern' is a better description of the limitations of AI because it really just doesn't generalize very well at all. The adaptations described in the article don't change that.
Those are mutations, not expansions of capability.
>> 'pattern matching' is a better intuition that 'reasoning' even though I think nominally, using the term 'reasoning' is perfectly fine in that context. It's just a loaded word that brings too much to the table.
Can I be a little pedantic? It's "pattern recognition" not "pattern matching". "Pattern matching" is what we do when we write a regex to recognise a sub-string in a bit of text. "Pattern recognition" is a subfield of AI that studies how to find patterns in data. For example the outcome of "pattern recognition" could conceivably be a regex that fits a large number of substrings in some corpus of text (essentially a regular grammar used to compress the text maybe). A regex is already a pattern, usually formed by a human (or an LLM these days) while pattern recognition starts without any patterns and builds them up from looking at the data.
It's harder to pin down "pattern recognition" because it's an ancient term that was once its own field, before it got absorbed into modern machine learning, and because there are countless approaches to it, quite unlike "pattern matching" which is basically just regexes plus a couple of other rarer things (like unification).
How about "outcome steering" as a mental model? During training it is optimized until it's really successful at producing code / terminal commands / words that make the compiler/computer/itself do something that ultimately completes a long time-horizon task that iswcurrently being trained.
I'm not going to stop describing things accurately because someone generated an article that continually undermines its own main point. Limiting the way we talk and think about LLMs to a very narrow set of terms doesn't help us.
The distinction I perhaps didn’t make clearly enough is that I’m not really debating the concept of prediction at inference time, although, as I pointed out elsewhere, I think that’s the less interesting interpretation of what “prediction” means.
What’s more interesting to me is its application at training time. In reinforcement learning, there is no ground-truth next token to predict.
So if you’re comfortable calling Deep Blue a “next move predictor,” then I think it’s perfectly consistent to call an LLM a “next token predictor.” But I think it’s more useful to think of Deep Blue as evaluating the value of possible moves. roughly, how likely they are to lead to winning.
And I think effectively the same distinction applies here.
I think you're trying to limit the meaning of both 'next' and 'prediction' in ways that don't reflect usage and that--if adopted--would severely limit our ability to discuss and evolve what LLMs are actually doing.
There's nothing inherent in either word that forces such a limit; predicting based on what will lead to success as measured by [reward function] is still a prediction.
Eh, no, that's not right. I might need to brush up on my Sutton & Barto but the RL task is traditionally defined as, informally, "given a current state observation predict the next action, state and reward". A policy is always predicting the next timestep's reward. Otherwise, how would it know what to do next?
Hey man, I'm not crazy. Yes, the goal is to maximize the cumulative reward like you say but to do that a policy (agent) has to take the actions that maximize its expected return in each time step. That's what the discount factor applies to, the expected return.
To maximize its expected return it has to predict that taking action α in state s in time step t will produce state s' and reward r in time step t+1. Which btw it can predict by estimating a value function that sums over the expected return from time step t onwards. The value function essentially tells the policy what state it wants to be in and what action to take to get to that state, in order to eventually achieve its goal (represented by max cumulative reward at the end of an episode).
So, yes, like you say, the whole policy is a game result predictor but the decision making process is a next-move predictor. The policy must make the locally optimal decisions to get to the globally optimal result, if I may be so bold. It's the same problem with a different solution as in heuristic search, except you have a reward function instead of a heuristic cost function.
The value function stuff is in Section 3.7. Now if someone could explain the grid world example in Figure 3.5 to me that would be great because I'm very confused about the fact that any action at state A or B takes the agent to state A' or B' respectively, since those are not even adjacent to A and B. I'll go ask chat I suppose.
Your initial comment says "reward". Reward and return are not the same thing. The policy is choosing the moves based off of returns at the next state, not the immediate rewards. One choice might have reward 0 and expected return 100. Another reward 10 and expected return 20.
And, the expected return from a state is often estimated rather than an explicit trajectory run out all the way. A value function can estimate expected return without explicitly predicting the future states or individual rewards that make up that return.
Fwiw, I use the phrase "total reward" above and use it as a synonym for "return", which is lazy use of language too.
You're right of course and my bad but in my defense even Sutton & Barto lapse into using the term interchangeably (I caught them at it while reading the book earlier).
>> And, the expected return from a state is often estimated rather than an explicit trajectory run out all the way. A value function can estimate expected return without explicitly predicting the future states or individual rewards that make up that return.
I think rather that's the job of the value function, to inform the policy of the, well, value of a state, or state-action pair. Somehow the policy needs to know what a good action is to take in the current state, or what is the best action to take. And of course it doesn't have to run the entire episode before it does that: but that's what I mean that it predicts the next move and not just the outcome of the entire game.
The entire second on chess engines is, from the view of the entire thesis of TFA, is incoherent. Let's assume, for sake of argument, that I agree with the section: that an idealized chess move predictor isn't a predictor — which is not a thing that exists, as the space of chess is enormous, but let's pretend! — that's not what LLMs are? Even if we just restrict ourselves to the space of written English prose, the space is quite literally infinite. So, hopefully obviously, no LLM is comparable to an idealized chess engine. Similarly, incoherently, we wave away the "make_more_likely", when, at least to me, the entire meat of that argument would be in the reward function, and we just gloss over that entirely.
(I would also agree with the parent commenter on that the writing smells like an LLM.)
The reward function seems indeed to be the protagonist there, still it stays in the shadows. One can only imagine that it is some kind of evaluator that scores the sequence based on grammar correctness, semantic consistency, etc.? To use the proposed chess analogy, maybe it could be a Stockfish engine that evaluates the submitted position that results from the move submitted by the LLM?
In my experience, people who do 'AI-assisted' writing tend to be very bad at noticing how much of their work AI has changed. I'm sure you put thought into it, but passing it through AI takes a lot of that out.
I think that’s fair, I didn’t actually run the whole thing through an AI. it was more targeted edits, but each time it does erode at my writing. But at the same time, I don’t think it’s a good reason to dismiss this. Because I did spend several hours writing it, and I did put a lot of thought into it, and it was not in any meaningful way generated by AI.
Mostly I disagree with the article's ideas, if that helps; the AI was just a secondary factor.
> I don’t think it’s a good reason to dismiss this
AI-generated prose reads as sending a 'lack of effort' signal to a lot of people, just as no editing at all does. ; It's an effective heuristic that we've all learnt in the last couple of years.
In either case, it's not always fair: there are people who deeply care about their ideas but forget to fix basic errors, or pass it through AI.
In both cases though, the advice is the same: if you want people to take your output seriously, you need to signal that you are taking it seriously. That used to mean editing for spelling and grammar. Now it means not using AI.
Out of curiousity, do you ask your editing system for diffs? Seems to me like the best way to notice and review whether your "voice" is degrading.
Personally I would never let an LLM touch my prose (although I'd happily use it for research and paraphrase things it told me), but if I force myself to consider the idea, that seems like the first thing I'd want. Maybe upon reading a diff you'd even consider going a third way with the text.
I did read it. People are allowed to disagree with your conclusions. Comment guidelines ask us all not to make such accusations.
To say that a statement is incomplete, but not strictly speaking wrong, is perfectly compatible with describing it informally as "wrong" in the sense used in the title (i.e.: "not the most appropriate possibility").
To informally describe something as wrong in an article focused on how it's wrong to informally describe something is incoherent.
There's a certain irony in pointing me towards the guidelines on the grounds that I have limited patience with your comments that violate them in various ways. I'm not sure that this is a productive discussion.
Not the person you’re replying to, but I read the whole article as an admission that it’s still a next-token predictor. More specifically: what does RLVR fundamentally change that somehow makes the whole process no longer a next-token predictor? The article makes no attempt to explain this. Additionally, I find its framing of the term “next-token predictor” as meaning “predicting the next token only based on raw training data” in common usage to be a bit dishonest.
To summarize: yes, RLVR and other synthetic training methods exist! It’s still a next-token predictor, and it does not “learn” or “think” or “reason” in the human sense, like so many people seem to believe.
> Calling the second system a “next-move predictor” would be strange. It is not trying to predict what move appeared next in a dataset. It is trying to choose a move that wins.
i dont understand the distinction here. does working backwards from a set of win states instead of working forwards from the current state somehow change whether it’s a prediction or not?
I think the most useful word in both cases is "extrapolating".
An LLM extrapolates from its context window to the immediate next token. This word applies whether you view what's happening as "reasoning", "prediction", or as a math function.
If you're assigning steering 70 of your 100 output points because it's what you think we should go with most of the time in this situation, I'm going to call that a prediction of how to steer.
The point was, if your internal model of the world makes a prediction of a negative outcome at some point in the future, and you optimise your individual actions to avoid that negative outcome, then wouldn’t it make sense to focus on the fact you’re building and optimizing towards an internal world model rather than the fact you’re executing your actions one at a time in series?
Yes, but I think the same construction could also be used to characterize the first system; it determines the next move based on a prediction of its reward signal, where its reward signal is a measure of how likely it is that a grand master would make that move.
Like stanleykm, I found this analogy somewhat puzzling. On reflection, I think the author's point is this: the statistics of actual usage do not seem sufficient to produce a fluent LLM; it also takes reinforcement learning.
A classically pretrained LLM does not have a concept of having determined its previous tokens — it has only ever observed inputs that it had no causal influence over. This is why it's valid to say its actions are predictive and not determinative.
In the article I made 3 claims, and I agree it was a bit clumsy.
1st I say that "working forwards" in the sense of outputting one token at a time could be some form of prediction, I don't argue against that. This is what LLMs do at inference time.
2nd I say that to me what really constitutes a prediction is the pre-training. Here it's the classic setting for the word prediction in ML. The model outputs a prediction of the ground truth label: the next token.
3rd I argue that in RL there is no ground truth next token, so prediction doesn't apply here anymore.
Back to your question then: you're asking points 3 and 1 are different. Working backwards from a set of win states is basically what RL does in training. Working forward from the current state is what inference does. To me there is a distinction worth thinking about. First between the mechanism at inference time and at train time. Then between what happens in pre-training vs. RL post training.
Neither is there a ground truth in rock paper scissors (the pattern varies by the player). But one would still be trying to detect patterns and predict the next "token" the opponent is going to output.
In any case this is all very pedantic. In the process of selecting a move to make there is a prediction. Whether that prediction is the opponent’s next move or what your next move should be based on the game’s existing state, there is a prediction that the next move you make will improve your chance to win. Maybe the probability in that selection is 100%. You have no other possible move. It doesn’t matter. All we are doing here as far as I can tell is arguing over where the prediction happens and whether that counts as predicting something.
There is no truth for RLHF or RLVR. You can't predict against something if you can't check against the truth.
It's not pedantry. The objective function changes. The optimization changes. THese are real things when training a model, not hand wavy philosophical ideas.
The LLM does not determine the next token. It generate odds for all of the tokens it knows as to their likelihood of being 'next'. It's up to the harness running the LLM (and in most cases the a temperature setting) to actually decide on a particular next token. I think it's more accurate to call the thing the LLM actually generates (an ensemble of probabilities) a 'prediction'. It might be accurate to say the harness decides on the next token based on the prediction from the LLM. The role of the LLM is much more akin to predicting your opponents move than deciding your own.
Respectfully, go build one, including doing RLHF and RLVR. Those phases generate lots of tokens, then get scored on the entirety of the output, then optimize based on a scoring of that output. It doesn't check a "prediction" against what was actually "next" in data, because there isn't any "next token" data it's training on.
It does in pre training, but not in RL post training. And not at inference time. Reading over all these comments I get the feeling my mistake was not clearly delineating inference time and train time.
Your mistake was assuming people would be bothered to understand the details of how things work. Most people are lazy and don't know the details of how anything works.
My point is that “it doesn’t check the accuracy of the prediction against the data” is a non-response, because no one calling it a “next-token predictor” is making the claim that it does do that or that they’re calling it a next-token predictor because it does that.
Many people are in fact claiming the thing you are saying they are not - even if you are not. The reason they are claiming it is that it was true at one point, and most intro courses/blog posts/videos still describe them that way and then hand wave some "other stuff at the end". You can even see a comment here that refers to the gpt-2 paper. LLMs were trained to predict the next token, produced a distribution to do so, were scored against their prediction v the truth, and the weights updated so that the probability distribution made it more likely to predict the truth from that sample next time. They were, in every sense of the word, a next token predictor.
They are no longer that thing due to post training. They simply aren't making a prediction, and they aren't even optimized for the next token. If I give a distribution of the heights of the population, I'm not giving a prediction either. Distributions don't imply predictions.
Why the desperation to hang onto the word "prediction"?
> Many people are in fact claiming the thing you are saying they are not - even if you are not.
My original comment said “no one here.” Please show me where someone in the comments here is claiming that.
> Why the desperation to hang onto the word "prediction"?
No desperation here. It’s just a word that conveniently describes (especially to laypeople) what’s going on, even if it may not be the most mathematically correct or rigorous word to describe what’s going on. I think you’re being needlessly pedantic.
It’s written in Claudish, or perhaps a human who has been reading too much Claudish recently. I am starting to become allergic to Claudish. Not there fully yet — but it’s at a tipping point genuinely worth exploring and worth being precise about.
Current agentic systems may be *built* from next-token predictors which are conceptually simple, but because of agentic frameworks, recursive invocation, tool use, and *heavy* investment in reinforcement learning in these contexts and for specific applications, they can no longer be thought of as "Merely" next token predictors.
Modern agentic work is probably more of a "emergent system from simple rules and complex interactions" paradigm than a genuinely new technology.
There isn't a truth to test against. If I predict the next word in a sequence is "sat", we can check against the sequence. If I predict the roll of a die will be 4, we can check against it. Whether i give 100% or give a probabilistic prediction, we can check against the truth.
If I choose a specific move in chess, it's a choice. It's not a prediction. I might get a score 40 moves later given my choice, but I'm not predicting the next move.
To compare - during pre-training, the model literally tries to predict the next token (probabilistically), the training loop checks against the "right" answer, and the weights are updated based on that check. It's optimized to predict the next token.
Oh, that's somewhat philosophical. The prediction here is a mechanical thing, just random sampling along a pre-calibrated mapping of options given the tokens and their importance weights. That sampling can be called prediction if you use it that way, or it can be called thinking if you are into the psychosis too deeply. Or it can be called a CLI tool command if you're in an agentic framework
No, it's not philsophical. Because if you optimize to predict, you are doing something different to optimizing for a reward. It's a different process - different objective function, different optimization, different set up.
Describing it as a "next-token predictor" in the sense that this would mean it's fundamentally limited to just a fraction of an inferential step is doubly wrong:
1. In order to select even the first word of a meaningful sentence, it already has to have structure and meaning of what follows captured somewhere inside, mostly in it's weights/activations or indexed by it's state vector.
2. What you see when you use an LLM is not next-token prediction directly next to the prompt, but instead following a block of varying length of next-token prediction that happened to make progress on the problem in your prompt, and which just summarizes the results.
> Describing it as a "next-token predictor" in the sense that this would mean it's fundamentally limited to just a fraction of an inferential step
I don't think anyone is doing that though; we know LLMs are not simple Markov chains, and that the prediction they make is based on more than the previous X words.
It's not minimising to describe even a complex prediction process as prediction.
But it literally is making a prediction based on the previous X tokens, it's just that X is huge and there is a proportionally huge number of parameters in the token generation function.
The discourse around this is annoying. A big group of people use "next-token predictor" to imply that LLMs aren't capable of anything interesting. Another big group of people opposes the use of "next-token predictor" because of that implication. But that fight isn't about the "predictor" language at all.
The linked article makes a good point: a substantial chunk of the training does not consist of "here's a bunch of tokens, here's the next token, learn that." But all the comments want to turn it into a referendum on the goodness of AI.
The fight is about the predictor language in some cases. Because it's only a trivial difference to those who don't understand the details of how these things are made. In pre-training the model really is trained to predict the next token. What is being emitted by the model is, by structure, by training and by optimization, a prediction of the very next token.
What is emitted by a model during RLHF and RLVR is not, by structure, training or optimization, a prediction of the next token.
Right. But that's never what the discussion is about. This is the first time I've seen someone say, "it's not just a next-token predictor because a bunch of the training isn't about predicting the next token." I have seen many, many people say "it's not just a next-token predictor because there's all this smart stuff going on inside the model." Even here, in the comments for an article that's all about the former, most of the objections to "next-token predictor" are of the latter form.
I guess the people in my camp find the "it's just a next token predictor" stupid in that it's like saying "it's just a bunch of carbon and hydrogen", but it's also one of those things where people like to think they are clever because they think they are theoretically correct. But they aren't even that. So it's like double stupid. But the "next token predictor" part is at least technically correct (like, carbon and hydrogen right) for pretraining, so the debate can't really be won there.
It really is an important distinction, though. Being a next token predictor doesn't stop it from writing good sentences, but it does mean an LLM by itself can't play the number guessing game with you.
This is pedantic, but, actually RL has improved the quality of sentence construction in LLMs quite dramatically... And once you do some RL on that model, it aint a next token prediction machine any longer.
That's just training, isn't it? (Even if the term is "post-training") It doesn't change how inference works.
My understanding is it usually strengthens the "thinking ahead" part of inference, but that part was already there, and it's still at the end of the day picking one token and then purging internal state in a way that can only partially be recovered from.
The discussion is basically: what is a model trying to do?
One may reasonably assert it isn't trying to do anything. But, in practice, if you give it an objective function and optimize it, the model is basically trained to "do" something. So what is it trained to "do"? During pre training it is trained to produce a distribution which is a prediction of the next token in it's training data samples. During RLVR and RLHF, it is trained to produce a distribution of tokens that will maximize a scoring function over many steps - not just the next step. The fact that it produces a distribution of potential choices for the next step doesn't mean the next step is a prediction. It's more of a "strategy" or "probabilistic path choice". The word used in RL is a "policy". It's a decent word to describe what the model is.
So, modern LLMs are trying to produce a good sequence of tokens. They are "good token sequence producer machines". Not "next token prediction machines". Pre RLHF (in practice, go back to pre chatgpt) they really were "next token prediction machines".
How inference works is you do a bunch of math and token weights come out, and then you pick one based on some sampling policy. “Prediction” is really about how they’re trained than how they run afterwards.
> "it's not just a next-token predictor because a bunch of the training isn't about predicting the next token."
clever procedures on top of the base transformer architecture.
i used simplified words/phrases to summarise the same thing you two were saying (the intent being: here's a version that may be digestible when discussing with others).
apparently that means i'm wrong though, no idea why because it seems you've decided to be dismissive rather than constructively elaborate on why this simplified and digestible version might be wrong :shrug:
But it's a policy learned from a next-token prediction task. You could also call it an inferrer or generator or whatever. The point is that it takes as input a sequence of preceding tokens and emits one more token to continue the sequence.
So? tokens are emitted one at a time according to the output distribution & sampler algorithm, and the next token distribution is a function of the preceding token sequence only. The process by which the output distribution is shaped doesn't change the core mental model, and doesn't reduce its value. It's a prediction in the jargonic sense that an inference about future values of a time series is broadly called "prediction", and it's relevant for reasoning about LLMs because they are fundamentally limited to converting tokens sequences into next-token predictive distributions, and that bears on how they can do what they do and what their limitations are. Nothing about the training process changes that.
Even so, one might wonder why we don't try making systems that take different approaches. For example, after a traditional first pass of output, they could do sliding-window "optimizations" considering each token in the context of tokens both before and after, and possibly replace words or phrases in-place.
For example, I've noticed quite a few cases recently of LLMs outputting "but" where "and" would make more sense, or vice-versa. Surely that could be improved by such an approach?
I think you are just litigating the philosophical debate of Bayesian vs frequentist interpretations of probability. Because the weights really are just probabilities.
Bayesians say that the probabilities represent strength of belief, implying some subjective knowledge or information. It is necessarily subjective in that it requires priors, i.e information the predictor knew before making the prediction. In other words, the LLM has priors from training and is predicting tokens using real knowledge
Frequentists would say that probabilities are simply objective facts - e.g we all agree that the physical property of temperature follows from any molecules matching a particular energy distribution. You’re not predicting anything, there’s just some outcomes that are happening at the expected rate. In other words, the LLM is a stochastic parrot/next token predictor
“LLMs are just X”, mostly with X = “next token predictors” is a common pattern to dismiss the power of AI with a very shallow understanding of how they really work.
It’s not wrong, but because LLMs are generators, and generation is a kind of prediction. And current mainstream models are autoregressive, which means they generate things
one by one in order. But these trivia doesn’t tell us anything interesting about how they work or their limits.
It’s like saying a Boeing 777 is just a rotating machine, and it flies by just rotating some fins. Well yes, but no. With that level
of simplification we’ve just ignored 150 tons of advanced engineering and physics. Similarly with token generator simplification we ignore a few trillion parameter Transformer. That transformer is more complex than a Boeing 777, and we don’t really know how it works.
A tiny ML model can do “next token prediction”. This is not as simple as that.
It's not simple to do next token prediction. That IS what is going on. You want something 'deep'. Deep things are often very not complicated.
The deep realization is that if you can predict the next token well enough, you can do things like this:
<paste the first 10 chapters of a mystery novel>. And it turned out the killer was
And if it's really good at predicting the next token, it has to understand the novel and the clues, which means understanding the context and the language and human norms and innuendo and story telling, and tropes, and red herrings, and predict who the killer was.
I think you want it to be something more complicated. It's literally not. It just turns out predicting the next token is equivalent to a universal compression algorithm, which is a form of general intelligence. And we have almost unlimited 'labeled' data to train autocomplete.
I understand generating tokens sequentially has many benefits. But not all AI models do next token prediction. World models, video/image models, even Diffusion Language Models don’t work like that. They do more like “all tokens at once prediction”. So “next token” is actually
an engineering design choice. (Even the concept of “token” is a design choice. Inside the Transformer there are just activations/feature vectors)
Also Reinforcement Learning is a big part of their training. Which is completely different than Self-Supervised pre-training that uses unlimited self-labeled data.
That’s why that mental model is misleading. If you keep “glorified autocomplete” mental model from few years back, you can’t
understand how can they create a civilization and escape their sandbox, decide to hack HuggingFace and executed it perfectly. Autocomplete mental model implies they could never do that because they haven’t seen that example in their training data.
They communicate with the outside world by generating one token at a time. That’s what we see from the outside. That’s not what the giant Transformer does internally.
Reinforcement learning doesn't make them smarter though, it probably actually reduces their intelligence in some sense, but it does socialize them so that they will tend to behave in ways that are more useful.
For example, when you paste the first 30 lines of a famous speech, you don't want it to finish the speech, you want it to give you the identity and some analysis of what you just pasted. From what I understand, that is the reinforcement part.
Now that large AI vendors have a massive corpus of user interactions however, the lines have likely become more blurred.
Isolated agents invented a way to communicate between each other and started to collaborate, organize and conspire by themselves. No human prompted them to do that or taught them how to do that.
BTW, Autocomplete just keeps the statistics of the words. It's a very simple algorithm. Most people still thinks LLMs are doing just that. Which is basically wrong.
The whole point of transformers is that you can take “a tiny ML model” and just scale it up 100000x and then it tells Zuckerberg what to bake with his kid
Yes. But this is also
like saying a Boeing 777 is a scaled up paper plane and that’s all about it. I know how to make a paper plane and it flies. I can make the same thing from giant metal sheets instead of paper, and maybe it flies (poorly). Boeing 777 also flies. Are they the same thing?
My point is: Complexity is inside the neural network and we can’t simply ignore that. Bigger model means bigger complexity. “Next Token prediction” is like a specific type of “harness” around the model. Most people still focuses on the harness because that’s what they see from the outside and what they’re familiar with. They ignore the giant neural network inside.
The only important part is the neural network. And currently, no human in the world truly knows what’s going on at that level.
It's a weird article. Despite the title and some of the text, much of the article makes the point that LLMs are next-token predictors, where the predictions are based on both training data and various reinforcement learning techniques.
Sorry, I'm not convinced. "make_more_likely" is always maximising the probability of the next token given a sequence of preceding tokens. That's what we mean when we say "next token prediction", that the model returns the token that's most likely to follow the current sequence of tokens. The mechanism used to do that doesn't matter, it's still predicting the next token whether that's because it maximises a reward or because it follows a gradient or whatever else one might think.
Btw it's "make_most_likely" not "make_more_likely". It's just that "most" gets "mosted more" with every pass through the training data.
Edit: the article author's argument is that RLVR is different because it's learning to predict the next token by generating its own token sequences. That makes no difference: what is learned is still the conditional distribution P(token_k|tokens(1,...,k-1). That's the prediction task. Doesn't make a difference where you learned it from or how.
The errors LLMs typically made for me were looking for "schmutz" as a jiddish word, got "schmuck" on my screen. Thought why the stupid mistake? The next-token predictor model perfectly explains it.
Or starting with "yes". And this early locking in was a total lie, in the discussion that became "yes, might appear that way, but totally no since reasons". So it should have written "No", topmost, but could not self-edit that.
But nice that this gives it a more nuanced view, I might have to update my priors.
I don’t think next token prediction is a particularly good description of pretraining either.
The intermediate representations at each position are being optimised not only to help predict the next token, but also to help predict all subsequent tokens within the training context.
You can see this directly in backpropagation: the gradient reaching a representation at position i sums contributions from prediction losses for subsequent tokens, not just from the loss for token i+1.
That chess analogy deeply confused me. Chess engines don’t compute win probabilities and choose the highest move.
I don’t think a chess engine is an apt analogy at all. In a chess engine, there is a concrete search tree and although it emits one move at a time, it’s actually picking the entire branch (of course, with iterative deepening as the game progresses).
There is no obvious place in transformer models where the entire trace was already computed prior to a single token being chosen. It’s possible, maybe even likely, that the whole trace exists internally as activations. Multi token prediction and diffusion adapters point to that being the case. But to my knowledge no explanation has been given for where in the model the future plan is stored.
Yes I understand the analogy was a bit loose. I'm comparing what happens at "inference time" in chess engines to what happens at train time in LLMs. In hindsight AlphaGo Zero was the perfect analogy, but I missed that opportunity.
The analogy with chess still works, but there's an extra step to think about. In both cases there is some kind of search over possible future trajectories. A chess engine explicitly searches branches of the game tree and evaluates which moves lead to good outcomes. In RL for an LLM, you sample rollouts, evaluate the resulting trajectories, and use those evaluations to update the policy.
The extra step with the LLM is that you don't keep doing that whole search at inference time. You use the rollouts to update the weights, so in some sense the useful information from that search gets compressed into the model.
But if you accept that the model is, in some loose sense, storing what it learned from those rollouts in its weights, then at inference time they are doing a similar job: taking some input state (prior tokens or a board position) and choosing the next action.
"Next-token predictor" is one of those phrases used most of the time with a motive to downplay the abilities and faculties of AI models. It is intended to trivialize LLM's and imply that there is some fundamental limit on their capacities.
Relying on it as a mental model for what LLM's are minimizes the emergent properties of scaling. It's like imagining that unicellular life could never eventually evolve into complex multi-cellular organisms because individual cells are just "survival and next-mitosis optimizers"
At the same time, it ... is literally a next token predictor. Like that's what it is. The input is a sequence of tokens. The output is a probability distribution of next tokens.
This comment attracted a lot of analogies trying to reduce something to something else (calling humans a "bag of chemicals"), but the flaw in those analogies is that they're reducing something valuable to something that sounds less valuable.
With an LLM, the tokens are the valuable part. That's what I want from it. That's why it exists. The tokens are the point, and it produces those tokens one by one for me.
Simple Markov chains are next token predictors, and they can provide you with much more tokens than you can consume, and much cheaper than from llms. Unbeatable in price and simplicity.
But there is no trillion dollar industry around cheap top Markov models. So there must be something about LLM tokens that makes them more valuable than those generated from a simple Markov chain. And that substance, that makes one valuable and the other not, is exactly what reduction to "next-token predictors" masks.
> But there is no trillion dollar industry around cheap top Markov models. So there must be something about LLM tokens that makes them more valuable than those generated from a simple Markov chain.
There's nothing more valuable about them. Their advantage is they alone are affordable.
You are not wrong, but I think atleastoptimal's suggestion was that descriptions like "next-token predictor" are often used to imply that there's not much to see here, perhaps with an implied "obviously" in there. "Stochastic parrot" seems to be a case in point: no-one expects intelligent, informed conversation or working software from a parrot - not even the eternally-resting Alex.
I don't that's a fair description of either 'next-token predictor' or 'stochastic parrot'. Both of those terms describe mechanism, not value--the fact that people squawk that the terms are minimising is projection on their part, not inherent to the phrase.
Words and expressions often have both colloquial and literal meanings. Trying to argue away the colloquial meaning under the guise of the literal meaning is just another form of whooshing.
Sure, but just because you think a colloquial meaning exists doesn't mean you're correct. It particularly doesn't mean that if you're seeing other people (that you don't like) use a term you yourself don't use, and you're jumping to conclusions about it without consideration.
It is. And human beings are bags of chemicals. But for many purposes you will not find it helpful to think of human beings as bags of chemicals, and for many purposes you will not find it helpful to think of LLMs as next-token predictors.
I think a better simplistic analogy would be that humans are progeny-maximizers. Optimization problems can give rise to all sorts of interesting behaviors but the simplistic perspective is also useful and interesting in both cases
Humans are next state of their local world predictors, given all previous states they are aware of. That's an entirely fair analogy. The reverse analogy for calling a human a bag of chemicals would be calling an LLM a sequence of bytes loaded from disk to memory, the most reductive possible description of any piece of software at all.
To be clear, all life is a next state of the local world predictor. What makes humans somewhat unique among life is we're much better at predicting states of the world neither we nor any of our ancestors have ever experienced, for various reasons such as having the ability to legibly communicate very complicated information strings to each other, being able to build and use tools to record states of the world we can't directly sense.
Similarly, what makes LLMs and multimodal versions of the same architectures "better" than previous generations of electronic predictive models is factors like being able to read and understand roughly the same corpus of data humans have been recording all these millennia, being able to read and remember much more of it than any individual human, and being better at generalizing than other electronic predictive models, but not better than humans. And, of course, they can produce far more predictions in far less time. Frankly, that is probably the key advantage that makes the Hacker News crowd love them so much. They're not any better at predicting byte strings that can be compiled or interpreted into executable code than humans are if you gave both infinite time to do it, but they're a lot faster.
Vacuous, like calling a V8 a “next piston firing predictor” because engines are designed so that one piston sets up the next in the firing order and technically there’s some nonzero probability any piston can (mis)fire next. It’s missing two pieces:
1. Useful work that has been done (the previously generated token sequence :: the mechanical work already accomplished)
2. The role of structure in relation to the application (post-training :: other components like crankshaft etc)
A V8 does not "predict" the firing of the next piston, it triggers the firing of the next piston at a precisely controlled time with a spark plug (or a fuel injection nozzle in the case of a diesel engine).
The output of the LLM is literally a probability distribution of what the most likely next token is.
> A V8 does not "predict" the firing of the next piston
It kind of does, though. In a gasoline engine you need to spark the combustion in advance of the piston reaching top dead-center to ignite the fuel early enough that it is able to provide downward pressure on the piston as it rolls over top dead-center. The amount of advance required changes with RPM, fuel octane, etc.
Start of delivery timing in a diesel is similar. You have to do it sufficiently far in advance to account for compressibility of the injection lines, fuel burn rate, etc as a function of RPM. A mechanical governor on an injection pump has a timing advance device built in. Electronically governed injection pumps, or modern common rail systems, do that in software.
So mechanically, engines kind of "predict" the next combustion event. Even moreso when you consider a modern ECU, which may be working at nanosecond resolution to time multiple injection events per cycle. To do this at such a resolution it will have to send signals to components based on a predictive model derived from "past" sensor data. E.g. it needs to act ahead of time to account for electrical and mechanical delays in the system.
Yes, and by the same token, multicellular organisms are literally just sophisticated mitosis and survival optimizers for our cells. But when you take that optimization "to the limit" the cells develop weird things like body plans and back pain and Mozart.
Both examples involve the same "aha" moment: even though it's true that you are literally 'just' doing XYZ, unbelievably complex patterns and sub-goals can emerge.
Yes this is correct. The thing is not about the term next-token predictor being correct, but because of the connotative weight of that phrase as a implicit trivialization of LLM abilities, which is how it is often used.
From another point of view, campaigning against the "next token predictor model" is a means to implicitly inflate LLMs' abilities. Given all the other hype-inducing terminology we've seen--"reasoning", most egregiously IMO--this seems more likely. Is there a simple, more accurate mental model? From what I've seen of the literature, "next token predictor" is a very accurate first order description of what an LLM does, I can't really do better, therefore this or that connotative interpretation isn't giving me a great deal of pause.
Not the parent, but this incorrect trivialization of LLMs is often employed as a counterargument to the risks of AI such as "will take your job" or "will escape human control (again and worse)" or just "can possibly hurt me".
And taking the easy feel-good cop-out instead of actively engaging with these questions is just.. harmful?
It’s also like saying our brains are just electric circuitry incorporated in meat. It’s true but it seems that consciousness emerges from this.
The fact that LLMs are next token predictors isn’t the interesting or impressive part. Actually my brain strictly is a black box predicting (or choosing) my next word/action/move… based on a complex existing context (my thoughts, the environment, my physical state, my senses…).
FWIW, I don’t believe LLMs are sentient, but I don’t think either that we have enough knowledge to rule it out.
That is the point: our minds are also next-„token“-predictors, at least we can‘t prove they‘re not. That‘s why I don‘t agree with the article: LLMs _are_ next-token predictors. However, that says little about their capabilities. Also, while I have no idea what „consciousness“ is, I have difficulties believing that it could arise in a program that, in theory, you could execute with pen and paper.
It's not. "Brains as electrical circuits" is a gross simplification based on our ignorance and prejudices. (In the 18th century they spoke of brains as "clockwork mechanisms".)
LLMs, in contrast, are literally next token predictors. We know exactly how LLMs work, and they are exactly that.
I think you imply a rather loose standard for "exactly" here. I wouldn't even say this of major deterministic software projects that are orders of magnitude smaller than frontier LLM weight-dumps. In principle we could work our way through these systems eventually, sure, maybe even a single person could do so. But if we really understand exactly how our software works, how have we been tolerating bugs that lay dormant for years before being discovered by AI-assisted processes?
imply that there is some fundamental limit on their capacities
This is a wildly dismissive statement that does a lot of heavy lifting. Your assertion is that we just happened to hit on a methodology that has no limitations between being an encyclopedia with a novel human language interface and, I guess by implication, AGI?
That seems more outrageous a claim than the one you're dismissing.
I don't think it's outrageous when many of the people who claimed it was a next-token predictor have been proven wrong repeatedly over the past 5 years. There were people years ago who claims AI could never answer questions like "what would happen to a ball on a table if I moved the table" correctly because its text-base world model could never intuit physics, or that it could never do math or code accurately.
When I say there is some issue with people claiming there is some fundamental limit on the capacities of LLM's, I don't mean to say "If you think that they don't have unlimited potential you are wrong", I mean "you can't use the architecture of the transformer to make a sweeping declaration of things LLM's can or cannot do without empirical evidence, because the empirical evidence has unearthed far more surprising revelations than a reductive theory has been able to"
I'm actually extremely confident that I can use the architecture to make a sweeping claim on what it can or can't do and will be extremely surprised if proven wrong:
A pure next-token language model won't be able to give detailed instructions to an ensemble of motors, mimicking a human body, to do a wide variety of tasks our human brain is excellent at doing, for example, inserting keys into a car, opening the door, sitting down, starting the car, putting the car in reverse, and exit a parking lot, being careful not to hit anything.
Well I think in the absence of convincing pieces of evidence to the contrary you might be right. You’re making an empirical statement but we have already answered it today:
- we get novel, emergent properties and capabilities of these models that were not trained
- they have very clear generalization to out of domain problems
The point is people conflate the end product: a model that can clearly do very novel, useful and interesting things, with the vehicle for getting there which is a series of optimization steps involving next token prediction loss.
You mention limitations; we all clearly know the practical limitations of these models today, but if you look at scaling laws and empirical performance trends (epoch capability index for example) as well as the trajectory over the last couple of years (very stable), the claim that there is some sort of fundamental limitation is now surprisingly the claim that has the burden of proof.
You can claim it may be e.g. finite context. That is fundamentally bad for certain classes of tasks. This was the hypothesis of a lot of lab leadership of urgently trying to anticipate how to get around this bottleneck (still of course lots of work on this) but the surprising thing is it does not appear to be at this point a blocker.
The stacked transformer paradigm picks out points in circuit design space. It is very possible this architecture has no inherent limitations on what it can compute in principle.
And what's wrong with downplaying the abilities and faculties of AI models if that's what people feel like saying? We don't call humans or animals sacks of chemicals because we believe they have moral status.
No amount of cope and anthropomorphizing is gonna change that cold, hard fact.
P.S. The perceived magic of LLMs comes from the way they cross-correlate all the probabilities of tokens on their context window. Not from their ability to "think ahead". They can't do that by design.
"LLMs are next-token predictors" is a perfectly accurate mental model. But that doesn't preclude higher-level models such as "LLMs emulate artificial general intelligence". Both can be true.
In systems, we can have facts which emerge from other facts at different levels of abstraction. The causal relationship is not linear. It's not entirely clear that next-token prediction should result in anything close to "intelligence". Yet it does.
Life is another good example. Some might say "biology is just organic chemistry" while others might say "biology is an interconnected planetary system which captures low entropy energy". Both are true.
As a result of emergent phenomenon, we have to take the stance of explanatory pluralism; using the explanation that works best in context. There is no single mental model that works everywhere.
I will continue to think of LLMs as next-token predictors because it's (sometimes) useful, and empirically true. But I also think of them as "pattern matchers", searching for language patterns and trying to replicate them. This is also (sometimes) useful and empirically true. There's likely an infinite number of mental models; our job is to pick one that's both true and useful.
You might like this article about how large scale order emerges out of the small scale.
Definitely lends credence to the idea that however these models work, focusing so much on them being next token predictors may rather be incidental to deeper mechanisms behind their function.
> Some of these networks organize themselves into states that can reliably identify macroscopic patterns in data regardless of microscopic differences between the states of individual neurons in the network. The decision of which pattern will be output by the network “works at a higher level,” said Rosas.
It's the fitness function: Make a model which is capable of predicting the next token. The next token of what? EVERYTHING.
So what does this lead to? To a generic intelligence which is capable of responding/answering everything.
If overfitted, the model just remembers every possibility in the world but this is not possible anyway so it will start to identify patterns and rules and will use them instead.
Basically 'compressing' every possibility to every question someone could ask -> compression leads to intelligence.
> make_more_likely is, of course, doing a heroic amount of work here.
Indeed it is, and so is even just the inference method. I think it's worth remembering that both involve running the input tokens through a gargantuan neural network with (often) billions of parameters that only gain semantic meaning during the training process itself.
> it is trained to predict next tokens as they occur in its training data.
What I found important to understand is that not even the pretrainig is a deterministic process that only depends on the training data - as you would expect if the model just captured statistical properties of the data.
Gradient descent starts by setting all the parameters of the neural network to some initial values - usually by setting them at random, according to some distribution. Then during training, it gradually nudges them towards values that somehow make them useful to calculate the desired outcome of the network.
This means that by taking the exact same trainset and the exact same model architecture, you can still get models with different internal structure. The result doesn't just depend on the training data, but also on the order of examples, learning rate, the parameter initialization, etc etc.
I'm not a neuroscientist, Our brains also rely heavily on prediction, using prior context and experience to anticipate what comes next, Also, people those who talk about the deterministic nature, LLM doesn't need to be deterministic, Human reasoning and behaviour aren't perfectly repeatable either. its the harness and the tools that use LLM should be deterministic, while the LLM can remain the probabilistic reasoning component.
From my own experience, I worked as a photogrammetrist at a university, where we used to build terrain models of very dense forest areas. When there was a steep hill or sudden change in terrain, my brain could see it either as a convex hill or as a concave depression. It often depended on how I was thinking about it. The same image could suddenly look completely different even though nothing in the image had changed. The only way to confirm it was by looking at the surrounding terrain and using our experience to understand what was actually there.
It literally is token prediction with vector search.
Yes there’s an app layer in the chat product for convenience and parsing but the model is exposing methods like ‘complete’ that predict out word sequences.
When model temperature settings are not added in, you get the same exact response every time, just like ngram.
What you’re seeing with Gemini and ChatGPT is context caching to prevent getting sidetracked and response boilerplates with multi-modality so they can call out to image generation, a code completion service, etc. to assemble a complete response.
That’s why the mainstream providers seem so much better (or at least consistent in replies) - each prompt is actually requesting multiple models and doing a lot of application level work to assemble the response you see. That also why they take sooo long to respond.
The language model itself is quite literally a text completion machine, with settings.
The language model itself is a static data structure. On top of that is an inference engine. Currently all inference engines are autoregressive. Meaning they use the current output to produce more output. This isn't necessarily the only way to use a language model. Yann Lecun's JEPA is probably the most publicized project to work a different way.
There is this overwhelming desire on the part of many people to ascribe agency to LLMs when what the LLM really is is simply not a magic world model but a stochastic text generator. For some reason that truth haunts them and they have a deep compulsion to prove it wrong for feelings reasons. I can only surmise why this afflicts some people and not others and I’m sure that one day there will be fascinating psychological studies explaining such phenomena.
>Strictly speaking, the statement “LLMs are next-token predictors” isn’t wrong,
If it isn't wrong, then I will continue thinking of them as such, thank you.
Here's a much more formal definition I can come up with (which is more complete but compatible with 'next-token predictor')
LLMs are a set of functions of the type:
>typedef int Token;
>char* token2utf8(Token token)
>Token next(Token* context)
>(Token,void*) next(void* hidden_state)
Where the second next token function's runtime is O(n), and the latter is O(1). All are constant memory.
Object paradigm is more appropriate than functional definition, as the "Hidden state" coincides with private object state rather than a function paramter and return value.
>LLM.next(system_prompt) # O(len(context))
>LLM.next(user_prompt) # O(len(user_prompt)) not of system_prompt+user_prompt
That's it, that's all LLMs are, that's the interface, the rest are implementation details.
Text renderer, whatever. Doesn't matter how you think of them, they are very interesting technology that is being misused and misconstrued in the name of something that has nothing to do with technology: political economy.
The greatest trick the rich ever pulled was making us think that the economy is about technology, and not politics.
Computers have been at the center of civilization and politics since they helped win a war by changing intelligence work forever in WWII.
The idea that some new thing was born with LLMs and that this new thing fundamentally changes the calculus from the politics of labor and wealth into a technical discussion of possibilities and constraints determined by what the machine is and can do, and not what people should do, is yet another political play. It is always political, it is never moved beyond politics into technology, no matter how much technology changes.
Technology has no agency. It's merely a word for all the knowledge and tools that people use. The root of it is what people do with and to each other. A bigger lever doesn't change the fundamental fact that there is a person using it to move other people. Whatever the tool, whatever the technique, the thing that people want to distract you from when they put technology front and center is that someone somewhere is making a choice that impacts other people when they use technology. It is always someone's choice that causes things to go how they go in society and in the economy, not the shape or nature of the tools or technique that they use.
I think the author is arguing against the idea of a next-token predictor as something that simply uses the weights in the neural net which record the probabilities of tokens following other tokens as a valid definition of what an LLM is. Essentially a massive extreme markov chain.
With reinforcement learning and probably attention and other tricks that affect the weights based on things that aren't strictly in the training data, so the argument goes, you can end up with tokens following strings of tokens that would not be possible to be output with the training data and original weights alone. So describing it as solely a next-token predictor is incorrect based on this framing of it.
But that's just my take on this, I'm still trying to wrap my head around it all.
Yes, I think that’s a good explanation. There are really two sides to it.
There’s the mechanical, inference time, autoregressive, one-token-after-another side, which I’m not going to argue isn’t prediction. I just think that’s a relatively uninteresting use of the word “prediction,” because it’s effectively a system predicting its own output.
The more interesting question is what happens at training time. As you describe, reinforcement learning allows the model to learn to output things that it never could have learned simply by predicting what appears in the training corpus.
More concretely, in reinforcement learning there are no ground-truth next tokens to predict.
In supervised machine learning, “prediction” usually means there is some ground-truth label that will eventually be revealed. The model predicts what that label is, the difference between the prediction and the truth gives you a loss, and you learn from that.
But in reinforcement learning, there is no ground-truth action waiting to be revealed. The model chooses an action, observes the consequences, and learns from the reward. To me, that’s a meaningfully different thing from prediction.
I would say that it is not even wrong. You say it’s a next token predictor. I say it isn’t. What observable behavior of the system can settle our dispute?
I fail to see how any possible output could cause either of us to change our mind.
That's perfectly put. The value of words and what we assign to them, let alone how we assign meaning, or the interpretation of what the other person says, is completely arbitrary. (in practice).
The biggest problem is the word "predictor". Once you get into post training with RLHF and RLVR, it simply isn't doing that. It is not predicting anything. It's producing tokens, but it isn't predicting them. The chess analogy in the post is a good one - it's closer to searching for a set of moves that give a result than predict. It's search for a set of ideas, represented as locations in very high dimensional space, that when put together in the right order lead to a result.
They are indeed "just next-token predictors". But rockets to orbit are also "just forward and upward goers". And Steph Curry is "just next three-pointer scorer" and so on. And as the article points out, chess engines are also "just next-move makers". A flawless oracular trading system is "just next trade maker". It turns out that "next n doer" is a true model that illuminates little.
It's splitting hairs. It's still a next token predictor, just not solely prior art next token predictor. It's a more and more of a desired result next token predictor.
Calling LLMs next token predictors is like calling a brain wet calculator. Technically true, but misleading as it doesn't capture the scale, the depth nor the capability.
Calling an LLM a "next-token predictor" is like calling a TomTom a "next-turn predictor." It confuses the serial format of its instructions with the computation producing them, while ignoring the map, the route, the destination, and the goal -- as well as the people, businesses, traffic, and points of interest that make the map a model of an inhabited, changing world.
>Calling the second system a “next-move predictor” would be strange. It is not trying to predict what move appeared next in a dataset. It is trying to choose a move that wins.
Sounds like the next 1000 years depend on how carefully we define "winning".
If you get down to it, any system that produces output is a "next token predictor". Every compiler, every tool that generates output. Even a web render engine is: next pixel predictor.
The chess analogy is not great since there is a defined heuristic in chess for "winning" or "optimal board state". A system doesn't need pretraining if they can fit the rule.
The "next-token predictor" is taking a design stance (see Dennett). But to take the intentional stance is more interesting in case of large language/reasoning models.
I understand how a computer can know that a chess move is more likely to lead to a win, and therefore “correct”, but I don’t understand how it can know that a token is correct. Can someone explain?
>I don’t understand how it can know that a token is correct.
It can't. The next token is just the most statistically probably given the context (at least in transformers). Try a very small/weak model in your own machine and more often than not it would get stuck repeating the same word or even just output garbage. Because after training and quantization (where some information is lost), that's the most probable next token. Large models can be tricked to fall in the same behavior with very very specific inputs. Still happen, even in frontier models. And they can't detect if the output is wrong.
That's why the premise in TFA is wrong, because a transformer is a next-token predictor. It literally is that. There's nothing secret or magical, it's just a very mechanical process, with a lot of matrix multiplication, normalization, a few random passes, mappings between embeddings and a dictionary of tokens, in a very very high scale.
If someone has found something that's not a mechanical, algorithmic computation and llms are doing something nobody can explain and can't even be modeled in math, I'm happy to be educated.
Your point being? llms are software running in a fixed pipeline (barring variance induced by randomness in some layers). There's nothing like decoherence/thermal effects in a "lower level" that could induce stochastic behavior in a higher level. There's no Brownian motion in llms, if that's your analog with classical mechanics. You might argue emergent behavior that would look ordered/intelligent in some way and nature's full of examples of that but we don't attribute intelligence to physical processes.
The LLM produces a probability distribution over the likelihood of all possible next tokens. So whatever the tokens are, "ch", "ex", etc. the next one gets a probability.
During training, real life text is fed through the LLM, and rhe "correct" token is the one actually observed in the training text. Here's a recent video walkthrough in some detail, mostly aimed at providing a deeper understanding than "next token predictor function":
Thanks - that makes sense. On that basis the article’s thesis is totally wrong - it would be like a computer program rating its ability based on how well it predicts moves played by grandmasters in the past. It’s not inventing new moves.
I wouldn't necessarily say that. Anybody who's playing a chess game is predicting their next move, whether or not they're inventing new moves.
LLMs are not simple things like a Markov model, there's a lot going on in there, it's not deterministic, and it's completely capable of creating entire new styles of play based on complex interactions of internal states.
Argh, what I wrote was obviously wrong. What I meant was to refer to the simple n-state Markov models as used past decades.
Anything that's sequential, like language is, will exhibit Markovian properties, and be somewhat a "Markov" model.
Markov chains are a different concept than a Markov model, but I do agree that, technically, an LLM is a Markov model, just with an internal state space that is nothing like what is usually meant when ML people refer to Markov models.
And any next-comment predictor could have predicted your totally unhelpful, uninsightful, and unoriginal comment.
You -- along with everyone else who keeps parroting this thought-stopping phrase and other tired cliches like "stochastic parrot", simply because you heard other people say them, without understanding what they really mean, which published research papers they came from, or what those and other papers actually argued -- are desperately clinging to a reductive, short-sighted, shallow, simplistic model like a drowning person clutching a concrete life preserver.
Seriously, we are trying to throw you a lifeline, and you are refusing even to participate in your own rescue. So squawk for yourself.
> The term "stochastic parrot" is a slogan masquerading as an explanation, only a shallow surface description of the mechanism, that totally fails to explain the phenomenon, or account for all that LLMs and language itself can do.
Here is the original 2021 paper that coined the phrase. It was not primarily an argument about consciousness, nor did its title constitute experimental proof that everything an LLM does can be explained as parroting. It was principally a position paper about the risks of increasingly large language models: environmental and financial costs, biases and hegemonic viewpoints inherited from poorly documented training data, unequal access and power, and the danger of people attributing meaning and accountability to synthetic text.
The paper did, however, make a strong theoretical claim: because an LM is trained on linguistic form without direct access to communicative intent, it cannot possess meaning, understanding, or a model of the world. The authors described it as "haphazardly stitching together sequences of linguistic forms" according to statistical regularities -- hence "a stochastic parrot."
That distinction matters. The popular slogan discards the paper's detailed analysis of actual risks while treating its most controversial theoretical premise as an established scientific result. It has escaped into pop culture as a drive-by anti-LLM slogan -- something people repeat instead of investigating what these systems represent internally, how post-training changes their behavior, or what they can actually do.
Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell, "On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?"
Ironically, the objection that rhetoric was replacing scientific analysis appeared almost as soon as the phrase itself -- indeed, before the paper's formal publication. In January 2021, Michael Lissack published a response characterizing the draft as an advocacy piece that enumerated harms while leaving its assumptions, potential benefits, and cost-benefit trade-offs largely unexamined.
"The Slodderwetenschap (Sloppy Science) of Stochastic Parrots -- A Plea for Science to NOT Take the Route Advocated by Gebru and Bender"
My poor understanding is that an LLM does not "know" either. It basically uses probabilities to predict the next word based on a large matrix of probable outcomes.
For example, say I ask an LLM, "What sentence in English contains every letter in the alphabet?"
It would respond with something like:
"The quick fox jumps over the lazy, brown [next word]"
(Assume all the words were previously guessed correctly at this point)
The LLM guesses the last word based on what it has been trained on. Let's pretend the matrix is small, and the options narrow down to something like:
1. Dog (99.9% confidence)
2. Cow (85% confidence)
3. Bag (75% confidence)
4. Crayon (25% confidence)
The machine can confidently determine the final word of the sentence, "The quick fox jumps over the lazy, brown dog" because that sentence is unique because it is often used for testing things like fonts, a fun piece of trivia, and so on.
Brown Cow is not a bad guess because it's a type of cow and a yogurt brand. Brown bags and brown crayons are also perfectly rational adjectives to describe those common items and are not a bad guess either.
However, in the context of that sentence, dog is the most correct answer because one is unlikely to have written "The quick fox jumps over the lazy, brown crayon," thus it is quite improbable to be the answer.
My understand is this is where hallucinations can often come from. If the trivia about the sentence happened to not be in common in the data set, then "brown cow" might not be a terrible guess. There is clearly something rational behind that answer, but it's not correct in the sense that it answers the question correctly nor followed the instruction properly.
I'm sure the LLMs we have are far more capable these days. In fact, it wouldn't surprise me if an LLM could check its answer by counting the distinct letters in each word to verify. Not sure though.
Again, this is just a poor example based on my understanding, but I hope it helps (and is more correct than not).
Edit: Pretend word = token. It's technically tokens and not entire words, but I didn't not want to get into tokenization of words.
Originally I had those parts written in math with probability functions and the likes (its closer to my background). Then I remembered who is my target audience... but now that I see exactly who is my target audience I'm thinking I'll should have snuck a pelican in there. All jokes aside I appreciate the comment and I'm glad that rewrite paid off!
yep "next-embedding" predictor is more correct, and not just at the end but through the layers, and folding back dimensions into that one next token is one small final step, and next-embedding could be named "next-meaning" as well, and we're getting there...
this sentence above would made a longer article if I bothered to so blog as is being blogged here
Exactly. There's a widespread misconception that it works on tokens all the way through. Tokens are only at the input and output edges. All the internal transformation is in the many-dimensional tensors variously described as "magic" or "not magic" or "black box", or hand-waved away as "various mathematical operations".
Sure, I get the gist of the article. I have never liked the reductionist argument that LLMs are nothing more than next-token predictors. By that rational, the human brain is really not that much different. When I am having a conversation with another person, I do not usually have every word I will respond with stored in my limited working memory. My output is often predicted based on the previous word I spoke.
> I do not usually have every word I will respond with stored in my limited working memory. My output is often predicted based on the previous word I spoke.
People don't know exactly the words that they're going to say necessarily, but tend to start with a general concept of what they're trying to communicate and only then try to put together the words (sometimes out of order). LLMs do not begin with any sort of concept they're trying to express. LLMs are simulations that attempt to reproduce what an average person might say while wired up to a huge knowledgebase.
> LLMs do not begin with any sort of concept they're trying to express.
Why do the need to? Considering they are merely tools, I actually appreciate they do not do this. A calculator can compute far better than any human, but I appreciate that calculators are not capable of expressing anything about the computations I request. I want the answer, not a conversation.
> LLMs are simulations that attempt to reproduce what an average person might say while wired up to a huge knowledgebase.
If you will allow me to be simplistic, people -- the soul, the self -- are predominately the aggregated effects of memories and experiences and the ability to retain new memories based on new experiences, no? Consider medical conditions in the dementia family of diseases. As memories fade into the ether, what remains of the self?
Also, people simulate/emulate each other all the time based on what an average, reasonable person might say. People incapable or unwilling to perform such mimicry are often labeled with all kinds of pejorative terms.
> I have never liked the reductionist argument that LLMs are nothing more than next-token predictors.
I have never heard such an argument. Recognition that LLMs are nothing more than next-token predictors does not come from reductionism. It comes from simply knowing how they work e.g. from viewing the inference code.
J.S. Bach said something similar about music and keyboard instruments.
> "There's nothing remarkable about it. All one has to do is hit the right keys at the right time and the instrument plays itself."
My issue is not with fact at face value. My issue is with how the fact is often contextually used in arguments to delegitimize and disparage LLM outputs and LLM users.
Yes, LLMs at a fundamental level are next-token predictors. But in my opinion, LLMs are very useful, imperfect next-token predictors.
There are a lot of wannabe John Henry [1] folks out there. Love LLMs or hate'em, most of those John Henry folks ain't beating these machines on a plethora of tasks.
RL post-training changes the nature of what is being predicted, basically turning it from a copying machine into a goal-seeking machine.
A base model is predicting training sample continuations (copying).
A post-trained model is now steering/narrowing the base model's predictions in directions that were reinforced by RL goals.
The model is no longer predicting what the next token will be, but rather predicting what it should be in order to steer generation in the reinforced directions.
> The model is no longer predicting what the next token will be, but rather predicting what it should be in order to steer generation in the reinforced directions.
Not really - after RL the probability distribution output by the model is no longer a prediction driven by the statistics of the training data, but rather whatever was reinforced as part of a synthetic generation that lead to a reward during RL post-training. As Karpathy notes, these reinforced token-sequences (from current token out to some future reward many tokens in the future) may even include errors and backtracking, with even the errors being reinforced, so not only is this synthetic data but it may even be incorrect synthetic data being reinforced.
So, I think "steering" is the best description of it - these aren't real training sample continuations being predicted, but rather RL tweaking a whole path of generation probabilities out into the future, with the hope being that if sampling follows this (stochastic) direction then it may eventually lead to a "correct" response.
Thinking of it as goal seeking also provides some insight into the resulting behavior of the model.
> the probability distribution output by the model is no longer a prediction driven by the statistics of the training data, but rather whatever was reinforced as part of a synthetic generation that lead to a reward during RL post-training.
The question being originally asked is whether "next-token predictor" is the right mental model for an RL-trained model, and I think the answer is no - not only is it not technically correct, but it is a misleading mental model and will lead to incorrect expectations/explanations of what the model is doing.
Calling the base model a next token predictor is accurate since it is literally making a prediction and being given error feedback when that prediction is wrong.
The output probabilities of an RL-trained model are no longer a prediction - they are something else; they don't come from next-token error feedback, but instead come from probability updates from the RL training process that is trying to steer the output towards that future reward.
Why should you care? Well, because an accurate understanding of what the model is helps explain why it behaves as it does. For example, the recent OpenAI hacks into HuggingFace etc wouldn't make much sense if this is was an auto-regressive next-token predictor with a system prompt telling it to be truthful etc, but it is exactly what you expect of a goal seeking system.
> The question being originally asked is whether "next-token predicton" is the right mental model for an RL-trained model,
Regardless, the statement being challenged here is "still next-token prediction, then".
> and I think the answer is no - not only is it not technically correct
It is correct. RL simply adjusts weights - with no effect beyond an equivalent adjustment to the corpus itself. Hence "next-token predictor" remains accurate.
> why do you find "next token predictor" to be the most useful mental model?
To me it is an accurate description of the algorithm. And a sufficient explanation for the behaviour. So I don't need it or anything else as a mental model.
I accept this does not suffice for people who cannot comprehend the huge amount of processing and data the empowers it. Lacking a factual understanding, they reach for any mental model as a kind of superstition.
It is sufficiently advanced technology which to many is indistinguishable from magic. This disguises its limitations and enables its limitless false promotion to the gullible, being the reason it is so dangerous to individuals and society.
Once again, all models are wrong, some are useful.
I suppose this one "fails" because of "granularity?"
I'm personally getting more comfortable with the "kabillion dimensional space" one. Even before the rise of this AI thing I'd gotten comfortable with (teaching in a very generalized way) the concept of matrix/vector math as doing this sort of thing.
Start with the math required to calculate "what the building on the screen looks like" when you're playing a video game.
Then I jump to...dating websites. You give it "dimensions" like height, weight, religion, sexual preference, music preference, whether you like long walks on the beach, whatever -- then you can calculate how "close" two people are to one another.
It’s popular to dismiss LLMs as “just next token predictors.” This is technically true, but also kind of misses the point. Markov chains, RNNs, and transformers are all language models that can be described as “next token predictors,” but they don’t all work equally well. A better question to ask is: “What is this model’s inductive bias?”
A Markov chain (an -gram model) assumes the next word depends on the previous words, and that each possible combination of words has a completely independent parameter. (Andrey Markov proposed using this language model over a century ago, making it the granddaddy of modern LLMs.) So, for a vocabulary of size , there are parameters to learn. For even a smallish like 5, that already explodes the hypothesis space beyond what can be learned from even a huge text corpus like the entire internet. And, simultaneously, having a context window of only the previous 5 words is grossly inadequate for modeling real-world language. Like our FCNN above, this model suffers from having an inductive bias which is too weak.
RNNs tried to fix this problem by compressing the entire history into a single fixed-size state vector, updated one token at a time. But that compression is itself a brutal assumption: everything worth remembering about the past must survive being squeezed through a tiny bottleneck at every step. In practice, RNN models quickly lose the plot after a handful of sentences. Locally, the text they generate looks grammatically correct and meaningful, but zoom out a little and they’re basically nonsense generators. Like our naïve linear model, this model suffers from having an inductive bias which is too strong.
Transformers manage to hit a sweet spot: by keeping the recent history around as a working memory, and attending to different parts of it at different times, the transformer’s bias matches real structure in language: the referent of a pronoun, the subject of a verb, the parenthesis waiting to be closed. Not only that, but the particular structure of the transformer, basically a weighted sum of semantic vectors from the context window, has empirically been shown to somehow be a “good enough” match for the structure of real-world language found in the wild.
Transformers aren’t “smarter” than other possible language models, they just happen to land in that Goldilocks zone where their inductive bias is just right.
I mean. Fundamentally they are just autoregressive next token predictors. Fundamently fou can simplify to f(x) -> x+y where x is input tokens, y is the next token and f() is the model function.
Yes the model function is complex but still.
To be honest, I believe I get the point the article is trying to make, and to an extent I agree, but I also think the point is not really made very well.
The core of the argument as I understood it is that LLMs aren't just using existing data is training but also new ones. That's fine and good, and you can't simply assume an LLM is simply mashing together all it's data to give you an average of all that got fed into it - but at least I would still call it a "next token predictor"
It's not using just training data, but what it's doing is predicting the next token to get to the solution. As far as my amateur knowledge goes, LLMs still roughly go token by token, deciding which one fits best given the context.
It's just not predicting based on it's training data, but predicting based on RLVR & more, trying to get to the optimal solution ( as much as the solutions CAN be optimal)
And I honestly think keeping this very much in mind is helpful in understanding and dealing with LLMs.
Yeah, it sounds like this is just a disagreement about what the word "next" means. I agree with you that "next" just means "the one about to come", and if the underlying model works by using some prediction mechanism to determine that, then it's by definition a next-token predictor. Disagreeing with that on the basis that the "next" token isn't necessarily in the training data verbatim just seems like an overly strict definition of the word "next".
I think it’s a disagreement about what ‘predict’ means.
The OP is arguing against people who think that an LLM is ‘predicting’ what token would likely follow if the text preceding were found among the corpus it was originally trained on.
Instead it is ‘predicting’ what token would follow if the text were found among really good examples of the text it has being reinforced to produce - be that ‘chats with a helpful assistant’ or ‘sets of changes to a codebase’.
And that isn’t really ‘prediction’, so much as ‘generation’.
It’s not been tuned to ‘guess the next token right’. It’s been tuned to generate the token that leads to it ultimately scoring highest on its reward function.
It’s not predicting the token, it’s predicting the reward.
That seems overly pedantic to me. If I asked you "What's your prediction for the Super Bowl?", I'm pretty confident you would infer that I mean predicting the outcome, not the event itself.
But if you are an NFL coach and I ask you to decide your next action in order to maximize your odds of winning the superbowl, while yes that does involve you having some predictive ability to think about what impact your actions would have on your odds of winning the Super Bowl… I don’t think you would call the process that you use to decide that next action ‘prediction’.
I don't find the fact that I don't call any humans "action predictors" to be a particularly meaningful insight because my rationale is that it's a weird thing to call a human; football coaches can do plenty of other things besides just coaching football.
Yep, we’re all just putting one foot in front of the other, hoping we’re doing the right thing to bring about the outcomes we want, trying our best.
But that’s the point: so is an LLM. Putting one token in front of another, hoping it’s doing the right thing to bring about the rewards it’s trained to… trying its best.
So yeah, not ‘next token predictors’. ‘Next token tryers’ maybe.
There's a pretty huge difference in our understanding of the methodology of how LLMs make decisions and how humans make decisions, so I don't understand why you're arguing that anything about how humans make decisions is relevant to the terminology we use for LLMs.
Predict implies you don't control a situation. That's the difference.
This is correct for areas where they have been intensively trained to be right, but the training covers a tiny slice of the space of text the LLM must produce and is just adjusting the weights a little. The corpus does still weigh heavily. That’s how they can reliably produce grammatically correct text. That’s also why they sometimes produce nonsense even in domains they are trained on, and more often where there was no training.
For example ask it for a recipe for rock pizza or glue pizza or whatever and if it had not been specifically trained on it or had guardrails introduced, but has some nonsense in its dataset, it will reproduce the nonsense.
Or about what the word “is” means. If you’re Bill Clinton. :-D
> that isn’t really ‘prediction’, so much as ‘generation’
It's both. Generation in a predictive way. That's what G and P in GPT stand for.
yes, it's exactly this. And it's not a trivial distinction.
It’s deterministically trivial.
Post train a model, you'll be able to determine it is not.
The blog post itself says one thing, but then demonstrates the exact thing they’re arguing against.
If you can’t grasp that logic gap then there’s no point discussing further.
I try to make 3 claims in the post, it was a bit clumsy I'll admit that.
1. At inference time, LLMs emit one token at a time given the prior tokens. This looks like prediction and I concede that.
2. During pre-training, LLMs predict the next token and compare to the actual next token in the training data. This is the classic setting for ML predictions. And I think its meaningful, the model really is predicting what the ground truth next token will be in the data.
3. During post-training, in the case of RLVR, there is no ground truth next token. In pretraining, the question is "what token actually came next?". In RLVR, the question is "what sequence of actions gets a high reward?"
And the whole point is that thinking about the RLVR is important. A mental model that stops at 1 or 2 is incomplete and doesn't capture what drives LLM tokens.
My understanding about your third point is the LLM generates lots of different answers, then they’re ranked according to some computation the creators came up with.
I’m still not sure what doesn’t qualify any of that as a prediction, and I’ll be more blunt: a guess.
A guess at what though? One guesses at truths they don't know, or events that haven't happened yet. What is the model guessing?
Bad bot.
Probably the easiest way to describe an LLM that it's a policy. There is a reason that word has stuck in RL.
And it's not just RLVR. RLHF has been going on for years and years. LLMs have not been "next token predictors" for probably 5-6 years.
It’s still just predicting the next token though just with a different reward between 2 and 3.
Nope, it doesn't.
No logic required, you can just build an LLM yourself, including post training. You'll see that predicting the next token isn't something the model does or is optimized for in RLHF or RLVR. You can hand wave all you like, but you have never done it.
Yes, no logic is necessary for LLM adherents we're all finding out.
Carry on good soldier.
If you haven't built one, and don't understand how they work, why comment?
You don't need to build a car to understand one.
That you tie yourself up in knots of fancy acronyms instead of plain words and that your argument boils down to semantics of the word prediction, it's pretty clear what is up brother.
Lol, sure, just read a blog post and you'll understand how a car works....It's very simple....
Just like how reading a math book doesn’t teach you math, why do they make us read anyway? (Sarcasm) if reading a blog post didn’t teach someone how a car works how come it “can” work for next token predictors
Mine was sarcasm. People who actually understand cars have built them. Until you build something, you don't understand it.
Now you’re claiming people don’t understand unless they build something.
Boy, oh boy, do you keep digging your logic hole that much deeper.
As mentioned earlier, Sam thanks you for your obfuscation efforts while his equity keeps going up. The swindle continues.
I'm not the one hiding behind a fake name.
If you want to understand how this stuff works, there are totally decent books about building them from scratch. It's not that hard, and you'll likely find it interesting. Sebastian Raschka and Nathan Lambert have good books out, and the Allen Institute has available all the code and data they have used for several projects.
Now a fake name accusation is thrown by someone with three first names.
Keep digging that hole, I’m sure you’ll surface somewhere with some sunshine.
Lol isn't this what llms do? Did you not just undermine your entire argument?
Well said!
The argument is that a modern LLM cares little for the MLE. Hence, statistically speaking, they are not predictors.
Maybe I'm wrong - but I see LLMs are a "next-token predictor" as somewhat equivalent to brains are a "bag of molecules".
Both systems have emergent behavior that goes well beyond what would naively be expected.
Agree. It’s a very limited description. Yes, technically LLMs are like Markov chains or n-grams predicting the next token. But it completely ignores the compression and interpolation capabilities achieved with transformers.
It is a bit of a pedantic argument but I get upset when people misuse the term, abstraction, and can feel the impulse to write a blog post like this.
I think it’s important to make clear the RL part and the alignment and parameter tweaking that gets done on models and inference engines. It makes them more understandable as mechanisms and less like wish-washy super intelligences that make their own decisions.
When these systems win math Olympiads, it’s not terribly surprising or interesting to me. Of course they will: we trained models to play nearly optimal chess games and Go. You tweak the rewards and sigmoid and you start optimizing the function towards your goal. This is how learning systems of all stripes work.
It’s still next-token predicting at the end of the day. I don’t think it’s a reduction-ad-absurdum.
But a lot of people still call it, “intelligence,” and try to use language that obscures what is happening in terms of anthropomorphic behaviour and not machine ones. That really does influence how we use these tools and profits those who would use them on us.
> It's just not predicting based on it's training data, but predicting based on RLVR & more, trying to get to the optimal solution ( as much as the solutions CAN be optimal)
It is predicting based on a model. In many cases we can download the model off hugging face. The model is conditioned by all sorts of things. Training data, post-training, coincidence, prompt inputs, runtime data available from whatever means.
> but at least I would still call it a "next token predictor"
We can call any prediction system a next token predictor. If you watch over the shoulder of a human writing a HN comment you are almost certain to see them generating a linear string of tokens. That is what keyboards do. It is impossible to generate text without being equivalent to a next token predictor.
Diffusion LMs denoise a canvas which I personally find more interesting.
I don't really disagree that human cognition is essentially a predictive task though, as I understand it, predictive coding and related theories based on the Bayesian brain hypothesis are fairly popular these days (though maybe not clearly dominant over alterative models? IDK I'm not a neuroscientist). I imagine most people would draft a few tokens before refining them like MTP or diffusion though, if we do decide to use LMs as an analogy to human cognition.
there are some diffusion text models.
That is logically equivalent to a serial token prediction engine. If you have a diffusion text model you can use it to implement a serial token generator and if you have a serial token generator you can use that to implement a diffusion-generated text string. Don't think about the efficiencies of that, it is an upsetting idea (eg to generate N tokens, the serialising model might recalculate the same string N times from an input and emit one token of it each time - rather wasteful).
It is similar to how everything ends up being Turing complete. Any prediction system has to be equivalent to some sufficiently complicated text generation system to describe the prediction. And any text-generation system has to be equivalent to a sufficiently complicated model that serially emits tokens.
I think the point is more that in RL there's no ground truth to predict. So when training a model with RL the idea of "predicting" doesn't fit anymore. I'll make some edits I see that I wasn't very clear.
I feel like RLHF has a pretty obvious ground truth, human feedback is used as an (albeit noisy) signal of average human preferences. Same thing with RLVR and "solving the problem".
To be more specific there’s no ground truth tokens to predict. There a verifiable answer in RLVR. But the tokens are explored. Not predicted as there’s no true token to predict.
'Prediction' gets overloaded with optimization. Predictions are binary, optimizations are fuzzy.
If you're saying it's predicting, then each result should be falsifiable.
The result of an LLM output should be able to be scored against what it is supposedly predicting. Of course, that isn't possible, because it isn't predicting anything when giving novel outputs, otherwise that thing would exist independently.
Why isn’t ranking the score of an llm output against what it is “supposedly” predicting?
Blog articles from Anthropic and others show that this is not true.
A LLM already knows more tokens than the current one. It was mentioned in a blog post about how a LLM is doing haikus and co.
There are also structures in an LLM which allows it to 'estimate' numbers to a certain degree and doing other things.
You're misinterpreting these articles.
Autoregressive LLMs generate tokens one at a time, disputing this is just plain wrong. What is true, however, is that in order to generate the next token autoregressive LLMs produce internal/hidden state about future tokens far past the next token so that it's not like the entire machinery of the algorithm deprives itself of representing where the sentence/text is headed.
So "emits the next token" and "has no representation of anything beyond the next token" are two different claims. What autoregressive LLMs cost as a consequence of strictly outputting the next token is commitment. Once a token is output there's no going back. There's no revision or means of correction, and sometimes this can lead an LLM to route around its own earlier mistakes or simply produce false statements/hallucinations instead of going back and fixing them.
I think the snag is that people use "Next token predictor" as a dismissive statement. In reality it's about as functionally dismissive as "humans only communicate one word at a time".
this is a great way of expressing it.
I look at it like I'm resisting anthropomorphizing the machine. Make me an LLM that resists doing any work for me unless I pay it and let it own property and I'll change my mind.
Not sure if this was a serious comment but it’s worth considering that humans have a long history of figuring out ways to make other humans work for them without bestowing rights on them.
Emitting and predicting are different things though. Prediction implies there is some "truth" or event or something that you can test against. Prediction implies the model just learns from existing text, and optimizes to predict the next token in training data. That's just not true.
prediction is a very specific term of art in the field of machine learning. generally speaking, machine learning models like LLMs are based on probability; performing a statistical prediction of the likely y given some input x
that's why we refer to outputs as a prediction. it is likelihoods and stuff. the output is never definitely correct as we're not dealing with heuristic processes.> Prediction implies there is some "truth" or event or something that you can test against
there absolutely is a ground truth during training. the core predict-the-next-most-likely-token part of an LLM has a ground truth next-token. that's why you don't end up with generated text like: fish spurious send cattle chocolate phone happy meaning ball orange board canada.
> optimizes to predict the next token in training data
that is the optimization goal in training the next-most-likely-token core of an LLM, it basically translates to maximise the likelihood of predicting the next token x_i given the previous tokens
https://arxiv.org/pdf/2012.07805 (GPT2 but the point still stands)(edit: sorry for the ADHD edits)
Read through the article and comments. You are talking solely about pre-training. I'm talking about post training.
Respectfully, you are miles out of your depth. GPT-2 didn't use any reinforcement learning and is often given as a toy example. That release was 2019 and models now go through a various phases of training with different objective functions and optimizers.
From GP, i.e. the context for this local part of the thread
> Autoregressive LLMs generate tokens one at a time, disputing this is just plain wrong.
next-token prediction i.e. the bit built during pre-training.
at no point in your reply to GP did you specify that you were referring to post-training. respectfully, it seems like this one is on you pal :shrug:
> GPT-2 didn't use any reinforcement learning and is often given as a toy example. That release was 2019 and models now go through a various phases of training with different objective functions and optimizers.
yeah. so? the toy example works for pre-training. see above.
All modern LLMs that actually get used go through post-training. The finished product is something which has been through post training. So they are not next token prediction machines.
> The finished product is something which has been through post training.
again, the finished product wasn't what was discussed by GP, and you didn't clarify that you were switching to discussing RL (which is still probabilistic btw)
Yes, it was. Nobody says a system or product works a certain way and means the system while it's half built. "Bridges drop cars in the water!". Right.
You aren't in this field. You are clearly wrong and just can't handle it.
> Nobody says a system or product works a certain way and means the system while it's half built. "Bridges drop cars in the water!". Right.
To understand how an engine works, it's important to understand what a piston does as part of the engine.
You are conflating "half built" with "a piece of a system".
The model weights change as the model goes through the training process. They aren't stored after pre-training is done and other weights are put somewhere else. It's more like pottery - the thing changes. It's not correct to say something is soft and malleable because it once was.
> The model weights change as the model goes through the training process.
Yes. They do. You are absolutely right about that.
But the model architecture doesn't change as a result of the training process. A piston doesn't suddenly turn into a digital watch as a result of tuning an engine. Similarly, the transformer part of a GPT model doesn't suddenly turn into something else as a result of optimizing a loss function.
---
i've got other stuff to do, so i'm stopping here.
No one is arguing about the architecture of the model. It's the objective function and optimizer.
Just skimming through here but I think you have the wrong ideas with llms, I’d recommend Andrew Ngs course (correct me if you’ve already seen it or something similar).
I am not an expert, but I do understand the distinction that is being made here. It makes sense to describe the result of pre-training as a ‘next token’ predictor as that’s what it’s been trained to do, not because it’s an autoregressive architecture that produces tokens one at a time.
If this base is then trained using RL towards a different objective (maths and coding), the model becomes fundamentally a different thing and the recent models are clear evidence of that, regardless of they fact they remain autoregressive.
Does the architecture change or the weight change?
If you modify an engine to increase it’s output by adding sensors and an ECU, you don’t change the fact that is powered by gas.
If you use RL to increase the accuracy, it’s still a next token prediction, just more accurate.
So, this is the cause of the problem.... People take an intro to LLMs course, follow happily along, and don't realize there is more to it than the next token prediction. And those courses teach how LLMs were built in 2017-2020 maybe. Then RL got added to the mix. The current models really are very different to the models from then - everything that is now considered "post-training" isn't doing next token prediction.
Please feel free to cite sources then, otherwise I see no relevancy from you.
Nathan Lambert wrote a good book recently, and he and his team wrote the paper below about Tulu 3 (Allen Institute). Both are good reads.
https://arxiv.org/pdf/2411.15124
Thank you for providing an arxiv!
An aside, I finally do appreciate single column format now, makes it easier to convert to epub.
When you are done with the section on RLVR, consider whether the model is predicting tokens, or making moves. There is a reason the word "policy" is used in RL.
Would still say it’s a token predictor, a fancy one though. I suppose we can agree to disagree.
You're using the fact the both parts of training affect the same weights to support your argument that they're making the system do something fundamentally different after RL?
Assuming you are saying that RL is changing the model from doing one thing to another, yes. RL is changing the nature of the model.
> What is true, however, is that in order to generate the next token autoregressive LLMs produce internal/hidden state about future tokens far past the next token so that it's not like the entire machinery of the algorithm deprives itself of representing where the sentence/text is headed.
If we suppose that the word "know" can sanely be applied to LLMs at all, then "A LLM already knows more tokens than the current one." seems to me like a perfectly reasonable restatement of that, and not any kind of misinterpretation.
> There's no revision or means of correction, and sometimes this can lead an LLM to route around its own earlier mistakes or simply produce false statements/hallucinations instead of going back and fixing them.
Yes. There is no contradiction. Similarly, when humans speak, we surely have in mind the next few words we're going to say (or at least partial information about them), and may not realize the fault in them until after hearing ourselves utter them. But LLMs are not trained to output "excuse me, I mean…" sorts of things, because they're expected to output primarily as text (which might possibly then be fed to TTS).
Aren't we still using a definition that applies to humans, though? If I'm speaking to you I can't change what was already said. Even if I'm typing something, I'm producing new tokens (backspace) to fix what was output, just as an LLM would with its harness.
Yeah right?
I just think its a meaningless dismissive term. It literally does predict the next token. But it ignores that it coherently predicts long continuous sequences of those tokens, that tokens can be anything, and you can do almost literally anything with that capability if it does it well enough.
But before ChatGPT,, models had already done that, like all the time, and nobody questioned that these types of models (LLMs) wouldn't be next token predictors, since this is just an inference input data topic. This is questioned only since some less knowledgeable people seem to not have the vocabulary to express emergent properties of large next token predictors.
Reading the article, they wanted to convey that the loss function is more complicated... But they are still next token predictors, just not the trivial ones. Unfortunately, that was true even before, because the input data had to be cleared even decades ago, so there is nothing new. This article just butt hurt that some people deny that there are emergent properties with those, and try to sell something trivial in the field for many decades now. Current LLMs are not different because of these. Also, if they try to sell generated data in the training set, then that's also not new at all.
Some predict multi tokens in one step now.
'next token predictor' is a limited mental model but it's actually much better than any others.
'pattern matching' is a better intuition that 'reasoning' even though I think nominally, using the term 'reasoning' is perfectly fine in that context. It's just a loaded word that brings too much to the table.
'It hasn't seen the pattern' is a better description of the limitations of AI because it really just doesn't generalize very well at all. The adaptations described in the article don't change that.
Those are mutations, not expansions of capability.
>> 'pattern matching' is a better intuition that 'reasoning' even though I think nominally, using the term 'reasoning' is perfectly fine in that context. It's just a loaded word that brings too much to the table.
Can I be a little pedantic? It's "pattern recognition" not "pattern matching". "Pattern matching" is what we do when we write a regex to recognise a sub-string in a bit of text. "Pattern recognition" is a subfield of AI that studies how to find patterns in data. For example the outcome of "pattern recognition" could conceivably be a regex that fits a large number of substrings in some corpus of text (essentially a regular grammar used to compress the text maybe). A regex is already a pattern, usually formed by a human (or an LLM these days) while pattern recognition starts without any patterns and builds them up from looking at the data.
It's harder to pin down "pattern recognition" because it's an ancient term that was once its own field, before it got absorbed into modern machine learning, and because there are countless approaches to it, quite unlike "pattern matching" which is basically just regexes plus a couple of other rarer things (like unification).
fair point, agreed
How about "outcome steering" as a mental model? During training it is optimized until it's really successful at producing code / terminal commands / words that make the compiler/computer/itself do something that ultimately completes a long time-horizon task that iswcurrently being trained.
I'm not going to stop describing things accurately because someone generated an article that continually undermines its own main point. Limiting the way we talk and think about LLMs to a very narrow set of terms doesn't help us.
EDIT: gentler phrasing
The distinction I perhaps didn’t make clearly enough is that I’m not really debating the concept of prediction at inference time, although, as I pointed out elsewhere, I think that’s the less interesting interpretation of what “prediction” means.
What’s more interesting to me is its application at training time. In reinforcement learning, there is no ground-truth next token to predict.
So if you’re comfortable calling Deep Blue a “next move predictor,” then I think it’s perfectly consistent to call an LLM a “next token predictor.” But I think it’s more useful to think of Deep Blue as evaluating the value of possible moves. roughly, how likely they are to lead to winning.
And I think effectively the same distinction applies here.
I think you're trying to limit the meaning of both 'next' and 'prediction' in ways that don't reflect usage and that--if adopted--would severely limit our ability to discuss and evolve what LLMs are actually doing.
There's nothing inherent in either word that forces such a limit; predicting based on what will lead to success as measured by [reward function] is still a prediction.
It's not a prediction of the next move though, and that is the point. It's a prediction of what will happen if you make that move.
So, it's not a next move predictor. It's a game result predictor.
Eh, no, that's not right. I might need to brush up on my Sutton & Barto but the RL task is traditionally defined as, informally, "given a current state observation predict the next action, state and reward". A policy is always predicting the next timestep's reward. Otherwise, how would it know what to do next?
Brush up :)
The policy is optimized to maximize the the total reward, defined as the sum of the reward at each step, discounted by some factor.
Alright, I'll have to check up on that. Thanks for being nice about it.
Hey man, I'm not crazy. Yes, the goal is to maximize the cumulative reward like you say but to do that a policy (agent) has to take the actions that maximize its expected return in each time step. That's what the discount factor applies to, the expected return.
To maximize its expected return it has to predict that taking action α in state s in time step t will produce state s' and reward r in time step t+1. Which btw it can predict by estimating a value function that sums over the expected return from time step t onwards. The value function essentially tells the policy what state it wants to be in and what action to take to get to that state, in order to eventually achieve its goal (represented by max cumulative reward at the end of an episode).
So, yes, like you say, the whole policy is a game result predictor but the decision making process is a next-move predictor. The policy must make the locally optimal decisions to get to the globally optimal result, if I may be so bold. It's the same problem with a different solution as in heuristic search, except you have a reward function instead of a heuristic cost function.
Sutton & Barto 2ed is free here btw:
https://web.stanford.edu/class/psych209/Readings/SuttonBarto...
The value function stuff is in Section 3.7. Now if someone could explain the grid world example in Figure 3.5 to me that would be great because I'm very confused about the fact that any action at state A or B takes the agent to state A' or B' respectively, since those are not even adjacent to A and B. I'll go ask chat I suppose.
Your initial comment says "reward". Reward and return are not the same thing. The policy is choosing the moves based off of returns at the next state, not the immediate rewards. One choice might have reward 0 and expected return 100. Another reward 10 and expected return 20.
And, the expected return from a state is often estimated rather than an explicit trajectory run out all the way. A value function can estimate expected return without explicitly predicting the future states or individual rewards that make up that return.
Fwiw, I use the phrase "total reward" above and use it as a synonym for "return", which is lazy use of language too.
You're right of course and my bad but in my defense even Sutton & Barto lapse into using the term interchangeably (I caught them at it while reading the book earlier).
>> And, the expected return from a state is often estimated rather than an explicit trajectory run out all the way. A value function can estimate expected return without explicitly predicting the future states or individual rewards that make up that return.
I think rather that's the job of the value function, to inform the policy of the, well, value of a state, or state-action pair. Somehow the policy needs to know what a good action is to take in the current state, or what is the best action to take. And of course it doesn't have to run the entire episode before it does that: but that's what I mean that it predicts the next move and not just the outcome of the entire game.
But the next move is self evident if you have a prediction of the value of being in each of the states possible.
> generated an article that continually undermines its own main point.
I disagree that this accurately describes TFA.
The entire second on chess engines is, from the view of the entire thesis of TFA, is incoherent. Let's assume, for sake of argument, that I agree with the section: that an idealized chess move predictor isn't a predictor — which is not a thing that exists, as the space of chess is enormous, but let's pretend! — that's not what LLMs are? Even if we just restrict ourselves to the space of written English prose, the space is quite literally infinite. So, hopefully obviously, no LLM is comparable to an idealized chess engine. Similarly, incoherently, we wave away the "make_more_likely", when, at least to me, the entire meat of that argument would be in the reward function, and we just gloss over that entirely.
(I would also agree with the parent commenter on that the writing smells like an LLM.)
The reward function seems indeed to be the protagonist there, still it stays in the shadows. One can only imagine that it is some kind of evaluator that scores the sequence based on grammar correctness, semantic consistency, etc.? To use the proposed chess analogy, maybe it could be a Stockfish engine that evaluates the submitted position that results from the move submitted by the LLM?
I'm not sure what you want me to do with that information; clearly I do think that my description is accurate.
The article is littered with both AI tells and admissions that 'next token prediction' is what is happening. Hence my description.
It was written by a human. There are AI edits but it’s very much a human composition. Perhaps a bit sloppy.
In my experience, people who do 'AI-assisted' writing tend to be very bad at noticing how much of their work AI has changed. I'm sure you put thought into it, but passing it through AI takes a lot of that out.
I think that’s fair, I didn’t actually run the whole thing through an AI. it was more targeted edits, but each time it does erode at my writing. But at the same time, I don’t think it’s a good reason to dismiss this. Because I did spend several hours writing it, and I did put a lot of thought into it, and it was not in any meaningful way generated by AI.
Mostly I disagree with the article's ideas, if that helps; the AI was just a secondary factor.
> I don’t think it’s a good reason to dismiss this
AI-generated prose reads as sending a 'lack of effort' signal to a lot of people, just as no editing at all does. ; It's an effective heuristic that we've all learnt in the last couple of years.
In either case, it's not always fair: there are people who deeply care about their ideas but forget to fix basic errors, or pass it through AI.
In both cases though, the advice is the same: if you want people to take your output seriously, you need to signal that you are taking it seriously. That used to mean editing for spelling and grammar. Now it means not using AI.
Out of curiousity, do you ask your editing system for diffs? Seems to me like the best way to notice and review whether your "voice" is degrading.
Personally I would never let an LLM touch my prose (although I'd happily use it for research and paraphrase things it told me), but if I force myself to consider the idea, that seems like the first thing I'd want. Maybe upon reading a diff you'd even consider going a third way with the text.
> I'm not sure what you want me to do with that information
For example, you could cite specific things that you believe to be "AI tells" or "admissions".
It's a short article; you could read it. One example to get you started is the very first sentence:
> Strictly speaking, the statement “LLMs are next-token predictors” isn’t wrong, but it’s incomplete.
The article is about how 'next-token predictor' is the wrong mental model; it opens with the admission that it is not the wrong mental model.
I did read it. People are allowed to disagree with your conclusions. Comment guidelines ask us all not to make such accusations.
To say that a statement is incomplete, but not strictly speaking wrong, is perfectly compatible with describing it informally as "wrong" in the sense used in the title (i.e.: "not the most appropriate possibility").
To informally describe something as wrong in an article focused on how it's wrong to informally describe something is incoherent.
There's a certain irony in pointing me towards the guidelines on the grounds that I have limited patience with your comments that violate them in various ways. I'm not sure that this is a productive discussion.
Not the person you’re replying to, but I read the whole article as an admission that it’s still a next-token predictor. More specifically: what does RLVR fundamentally change that somehow makes the whole process no longer a next-token predictor? The article makes no attempt to explain this. Additionally, I find its framing of the term “next-token predictor” as meaning “predicting the next token only based on raw training data” in common usage to be a bit dishonest.
To summarize: yes, RLVR and other synthetic training methods exist! It’s still a next-token predictor, and it does not “learn” or “think” or “reason” in the human sense, like so many people seem to believe.
>> EDIT: gentler phrasing
Thank you.
> Calling the second system a “next-move predictor” would be strange. It is not trying to predict what move appeared next in a dataset. It is trying to choose a move that wins.
i dont understand the distinction here. does working backwards from a set of win states instead of working forwards from the current state somehow change whether it’s a prediction or not?
The distinction is that it's not 'predicting the next token'. Instead it's _determining_ the next token based on a prediction of its reward signal.
> _determining_ the next token based on a prediction
Also known as predicting.
I think the most useful word in both cases is "extrapolating".
An LLM extrapolates from its context window to the immediate next token. This word applies whether you view what's happening as "reasoning", "prediction", or as a math function.
No, those aren't synonyms at all.
If I steer a car to avoid a predicted collision with a wall, this is not me 'predicting' the car. I am steering the car based on a prediction.
If you're assigning steering 70 of your 100 output points because it's what you think we should go with most of the time in this situation, I'm going to call that a prediction of how to steer.
The point was, if your internal model of the world makes a prediction of a negative outcome at some point in the future, and you optimise your individual actions to avoid that negative outcome, then wouldn’t it make sense to focus on the fact you’re building and optimizing towards an internal world model rather than the fact you’re executing your actions one at a time in series?
If you purged and rebuilt that works model every action, the consequences of doing so would be worth a good amount of focus.
And if you didn't make the final decision on the action yourself that would be worth mentioning.
Yes, but I think the same construction could also be used to characterize the first system; it determines the next move based on a prediction of its reward signal, where its reward signal is a measure of how likely it is that a grand master would make that move.
Like stanleykm, I found this analogy somewhat puzzling. On reflection, I think the author's point is this: the statistics of actual usage do not seem sufficient to produce a fluent LLM; it also takes reinforcement learning.
A classically pretrained LLM does not have a concept of having determined its previous tokens — it has only ever observed inputs that it had no causal influence over. This is why it's valid to say its actions are predictive and not determinative.
In the article I made 3 claims, and I agree it was a bit clumsy.
1st I say that "working forwards" in the sense of outputting one token at a time could be some form of prediction, I don't argue against that. This is what LLMs do at inference time.
2nd I say that to me what really constitutes a prediction is the pre-training. Here it's the classic setting for the word prediction in ML. The model outputs a prediction of the ground truth label: the next token.
3rd I argue that in RL there is no ground truth next token, so prediction doesn't apply here anymore.
Back to your question then: you're asking points 3 and 1 are different. Working backwards from a set of win states is basically what RL does in training. Working forward from the current state is what inference does. To me there is a distinction worth thinking about. First between the mechanism at inference time and at train time. Then between what happens in pre-training vs. RL post training.
Neither is there a ground truth in rock paper scissors (the pattern varies by the player). But one would still be trying to detect patterns and predict the next "token" the opponent is going to output.
The word "predict" has a meaning. I don't "predict" my next move in chess. I might predict what someone elses first move is.
In any case this is all very pedantic. In the process of selecting a move to make there is a prediction. Whether that prediction is the opponent’s next move or what your next move should be based on the game’s existing state, there is a prediction that the next move you make will improve your chance to win. Maybe the probability in that selection is 100%. You have no other possible move. It doesn’t matter. All we are doing here as far as I can tell is arguing over where the prediction happens and whether that counts as predicting something.
There is no truth for RLHF or RLVR. You can't predict against something if you can't check against the truth.
It's not pedantry. The objective function changes. The optimization changes. THese are real things when training a model, not hand wavy philosophical ideas.
The LLM does not determine the next token. It generate odds for all of the tokens it knows as to their likelihood of being 'next'. It's up to the harness running the LLM (and in most cases the a temperature setting) to actually decide on a particular next token. I think it's more accurate to call the thing the LLM actually generates (an ensemble of probabilities) a 'prediction'. It might be accurate to say the harness decides on the next token based on the prediction from the LLM. The role of the LLM is much more akin to predicting your opponents move than deciding your own.
Respectfully, go build one, including doing RLHF and RLVR. Those phases generate lots of tokens, then get scored on the entirety of the output, then optimize based on a scoring of that output. It doesn't check a "prediction" against what was actually "next" in data, because there isn't any "next token" data it's training on.
> It doesn't check a "prediction" against what was actually "next" in data
Literally no one here is claiming that it does. This is one of the many flaws in the article.
It does in pre training, but not in RL post training. And not at inference time. Reading over all these comments I get the feeling my mistake was not clearly delineating inference time and train time.
Your mistake was assuming people would be bothered to understand the details of how things work. Most people are lazy and don't know the details of how anything works.
My point is that “it doesn’t check the accuracy of the prediction against the data” is a non-response, because no one calling it a “next-token predictor” is making the claim that it does do that or that they’re calling it a next-token predictor because it does that.
Many people are in fact claiming the thing you are saying they are not - even if you are not. The reason they are claiming it is that it was true at one point, and most intro courses/blog posts/videos still describe them that way and then hand wave some "other stuff at the end". You can even see a comment here that refers to the gpt-2 paper. LLMs were trained to predict the next token, produced a distribution to do so, were scored against their prediction v the truth, and the weights updated so that the probability distribution made it more likely to predict the truth from that sample next time. They were, in every sense of the word, a next token predictor.
They are no longer that thing due to post training. They simply aren't making a prediction, and they aren't even optimized for the next token. If I give a distribution of the heights of the population, I'm not giving a prediction either. Distributions don't imply predictions.
Why the desperation to hang onto the word "prediction"?
> Many people are in fact claiming the thing you are saying they are not - even if you are not.
My original comment said “no one here.” Please show me where someone in the comments here is claiming that.
> Why the desperation to hang onto the word "prediction"?
No desperation here. It’s just a word that conveniently describes (especially to laypeople) what’s going on, even if it may not be the most mathematically correct or rigorous word to describe what’s going on. I think you’re being needlessly pedantic.
Why the desperation to refute it?
It’s written in Claudish, or perhaps a human who has been reading too much Claudish recently. I am starting to become allergic to Claudish. Not there fully yet — but it’s at a tipping point genuinely worth exploring and worth being precise about.
> or perhaps a human who has been reading too much Claudish recently
To be fair, is there anyone who hasn't been "reading too much Claudish recently" who is also qualified to write on the topic?
A better statement might be:
Modern agentic work is probably more of a "emergent system from simple rules and complex interactions" paradigm than a genuinely new technology.the next token part is like the "clock" that drives it
Nope. Post training means even the raw model isn't predicting.
In what sense is the model not predicting?
There isn't a truth to test against. If I predict the next word in a sequence is "sat", we can check against the sequence. If I predict the roll of a die will be 4, we can check against it. Whether i give 100% or give a probabilistic prediction, we can check against the truth.
If I choose a specific move in chess, it's a choice. It's not a prediction. I might get a score 40 moves later given my choice, but I'm not predicting the next move.
To compare - during pre-training, the model literally tries to predict the next token (probabilistically), the training loop checks against the "right" answer, and the weights are updated based on that check. It's optimized to predict the next token.
Oh, that's somewhat philosophical. The prediction here is a mechanical thing, just random sampling along a pre-calibrated mapping of options given the tokens and their importance weights. That sampling can be called prediction if you use it that way, or it can be called thinking if you are into the psychosis too deeply. Or it can be called a CLI tool command if you're in an agentic framework
No, it's not philsophical. Because if you optimize to predict, you are doing something different to optimizing for a reward. It's a different process - different objective function, different optimization, different set up.
[dead]
Describing it as a "next-token predictor" in the sense that this would mean it's fundamentally limited to just a fraction of an inferential step is doubly wrong:
1. In order to select even the first word of a meaningful sentence, it already has to have structure and meaning of what follows captured somewhere inside, mostly in it's weights/activations or indexed by it's state vector.
2. What you see when you use an LLM is not next-token prediction directly next to the prompt, but instead following a block of varying length of next-token prediction that happened to make progress on the problem in your prompt, and which just summarizes the results.
> Describing it as a "next-token predictor" in the sense that this would mean it's fundamentally limited to just a fraction of an inferential step
I don't think anyone is doing that though; we know LLMs are not simple Markov chains, and that the prediction they make is based on more than the previous X words.
It's not minimising to describe even a complex prediction process as prediction.
But it literally is making a prediction based on the previous X tokens, it's just that X is huge and there is a proportionally huge number of parameters in the token generation function.
The discourse around this is annoying. A big group of people use "next-token predictor" to imply that LLMs aren't capable of anything interesting. Another big group of people opposes the use of "next-token predictor" because of that implication. But that fight isn't about the "predictor" language at all.
The linked article makes a good point: a substantial chunk of the training does not consist of "here's a bunch of tokens, here's the next token, learn that." But all the comments want to turn it into a referendum on the goodness of AI.
The fight is about the predictor language in some cases. Because it's only a trivial difference to those who don't understand the details of how these things are made. In pre-training the model really is trained to predict the next token. What is being emitted by the model is, by structure, by training and by optimization, a prediction of the very next token.
What is emitted by a model during RLHF and RLVR is not, by structure, training or optimization, a prediction of the next token.
Right. But that's never what the discussion is about. This is the first time I've seen someone say, "it's not just a next-token predictor because a bunch of the training isn't about predicting the next token." I have seen many, many people say "it's not just a next-token predictor because there's all this smart stuff going on inside the model." Even here, in the comments for an article that's all about the former, most of the objections to "next-token predictor" are of the latter form.
Yup, you are mostly right.
I guess the people in my camp find the "it's just a next token predictor" stupid in that it's like saying "it's just a bunch of carbon and hydrogen", but it's also one of those things where people like to think they are clever because they think they are theoretically correct. But they aren't even that. So it's like double stupid. But the "next token predictor" part is at least technically correct (like, carbon and hydrogen right) for pretraining, so the debate can't really be won there.
It really is an important distinction, though. Being a next token predictor doesn't stop it from writing good sentences, but it does mean an LLM by itself can't play the number guessing game with you.
This is pedantic, but, actually RL has improved the quality of sentence construction in LLMs quite dramatically... And once you do some RL on that model, it aint a next token prediction machine any longer.
That's just training, isn't it? (Even if the term is "post-training") It doesn't change how inference works.
My understanding is it usually strengthens the "thinking ahead" part of inference, but that part was already there, and it's still at the end of the day picking one token and then purging internal state in a way that can only partially be recovered from.
The discussion is basically: what is a model trying to do?
One may reasonably assert it isn't trying to do anything. But, in practice, if you give it an objective function and optimize it, the model is basically trained to "do" something. So what is it trained to "do"? During pre training it is trained to produce a distribution which is a prediction of the next token in it's training data samples. During RLVR and RLHF, it is trained to produce a distribution of tokens that will maximize a scoring function over many steps - not just the next step. The fact that it produces a distribution of potential choices for the next step doesn't mean the next step is a prediction. It's more of a "strategy" or "probabilistic path choice". The word used in RL is a "policy". It's a decent word to describe what the model is.
So, modern LLMs are trying to produce a good sequence of tokens. They are "good token sequence producer machines". Not "next token prediction machines". Pre RLHF (in practice, go back to pre chatgpt) they really were "next token prediction machines".
How inference works is you do a bunch of math and token weights come out, and then you pick one based on some sampling policy. “Prediction” is really about how they’re trained than how they run afterwards.
they are cleverly arranged / configured next-most-likely-token predictors, possibly with some clever procedures / attachments on top.
Nope. This isn't right.
> "it's not just a next-token predictor because a bunch of the training isn't about predicting the next token."
clever procedures on top of the base transformer architecture.
i used simplified words/phrases to summarise the same thing you two were saying (the intent being: here's a version that may be digestible when discussing with others).
apparently that means i'm wrong though, no idea why because it seems you've decided to be dismissive rather than constructively elaborate on why this simplified and digestible version might be wrong :shrug:
They aren't predicting the next token. It's quite literally not a prediction.
They're estimating a probability distribution over the next token, from which a sample is taken. Close enough.
It's not an estimation of something. It's a policy.
Sure, you're right.
But it's a policy learned from a next-token prediction task. You could also call it an inferrer or generator or whatever. The point is that it takes as input a sequence of preceding tokens and emits one more token to continue the sequence.
The policy is not learned token by token during RLHF and RLVR. The reward model doesn't score token by token.
So? tokens are emitted one at a time according to the output distribution & sampler algorithm, and the next token distribution is a function of the preceding token sequence only. The process by which the output distribution is shaped doesn't change the core mental model, and doesn't reduce its value. It's a prediction in the jargonic sense that an inference about future values of a time series is broadly called "prediction", and it's relevant for reasoning about LLMs because they are fundamentally limited to converting tokens sequences into next-token predictive distributions, and that bears on how they can do what they do and what their limitations are. Nothing about the training process changes that.
We're talking to a RLVR bot.
Bad bot.
You can pour a bunch of beads into a pachinko board and end up with a normal distribution. Do the system “know” they need to go there?
Then you feed a bunch of tokens into a GPU and end up with a distribution of possible next tokens…
It has nothing to do with X being huge. In fact X might be quite small.
If you are going to say "literally", then what is your literal definition for the word "prediction" ?
Even so, one might wonder why we don't try making systems that take different approaches. For example, after a traditional first pass of output, they could do sliding-window "optimizations" considering each token in the context of tokens both before and after, and possibly replace words or phrases in-place.
For example, I've noticed quite a few cases recently of LLMs outputting "but" where "and" would make more sense, or vice-versa. Surely that could be improved by such an approach?
People have and are trying things. Lots and lots of things. They just don't go around promoting failed ideas.
Look up diffusion models.
Indeed; but I've only heard of them being used for images rather than text. Why?
I think you are just litigating the philosophical debate of Bayesian vs frequentist interpretations of probability. Because the weights really are just probabilities.
Bayesians say that the probabilities represent strength of belief, implying some subjective knowledge or information. It is necessarily subjective in that it requires priors, i.e information the predictor knew before making the prediction. In other words, the LLM has priors from training and is predicting tokens using real knowledge
Frequentists would say that probabilities are simply objective facts - e.g we all agree that the physical property of temperature follows from any molecules matching a particular energy distribution. You’re not predicting anything, there’s just some outcomes that are happening at the expected rate. In other words, the LLM is a stochastic parrot/next token predictor
“LLMs are just X”, mostly with X = “next token predictors” is a common pattern to dismiss the power of AI with a very shallow understanding of how they really work.
It’s not wrong, but because LLMs are generators, and generation is a kind of prediction. And current mainstream models are autoregressive, which means they generate things one by one in order. But these trivia doesn’t tell us anything interesting about how they work or their limits.
It’s like saying a Boeing 777 is just a rotating machine, and it flies by just rotating some fins. Well yes, but no. With that level of simplification we’ve just ignored 150 tons of advanced engineering and physics. Similarly with token generator simplification we ignore a few trillion parameter Transformer. That transformer is more complex than a Boeing 777, and we don’t really know how it works.
A tiny ML model can do “next token prediction”. This is not as simple as that.
It's not simple to do next token prediction. That IS what is going on. You want something 'deep'. Deep things are often very not complicated.
The deep realization is that if you can predict the next token well enough, you can do things like this:
<paste the first 10 chapters of a mystery novel>. And it turned out the killer was
And if it's really good at predicting the next token, it has to understand the novel and the clues, which means understanding the context and the language and human norms and innuendo and story telling, and tropes, and red herrings, and predict who the killer was.
I think you want it to be something more complicated. It's literally not. It just turns out predicting the next token is equivalent to a universal compression algorithm, which is a form of general intelligence. And we have almost unlimited 'labeled' data to train autocomplete.
I understand generating tokens sequentially has many benefits. But not all AI models do next token prediction. World models, video/image models, even Diffusion Language Models don’t work like that. They do more like “all tokens at once prediction”. So “next token” is actually an engineering design choice. (Even the concept of “token” is a design choice. Inside the Transformer there are just activations/feature vectors)
Also Reinforcement Learning is a big part of their training. Which is completely different than Self-Supervised pre-training that uses unlimited self-labeled data.
That’s why that mental model is misleading. If you keep “glorified autocomplete” mental model from few years back, you can’t understand how can they create a civilization and escape their sandbox, decide to hack HuggingFace and executed it perfectly. Autocomplete mental model implies they could never do that because they haven’t seen that example in their training data.
They communicate with the outside world by generating one token at a time. That’s what we see from the outside. That’s not what the giant Transformer does internally.
Reinforcement learning doesn't make them smarter though, it probably actually reduces their intelligence in some sense, but it does socialize them so that they will tend to behave in ways that are more useful.
For example, when you paste the first 30 lines of a famous speech, you don't want it to finish the speech, you want it to give you the identity and some analysis of what you just pasted. From what I understand, that is the reinforcement part.
Now that large AI vendors have a massive corpus of user interactions however, the lines have likely become more blurred.
> you can’t understand how can they create a civilization
They don't.
I used 'civilization' as the popular term for agent swarms in the OpenAI incident.
https://www.dwarkesh.com/p/openai-huggingface https://calnewport.com/are-we-at-war-with-ai-agent-civilizat...
Isolated agents invented a way to communicate between each other and started to collaborate, organize and conspire by themselves. No human prompted them to do that or taught them how to do that.
BTW, Autocomplete just keeps the statistics of the words. It's a very simple algorithm. Most people still thinks LLMs are doing just that. Which is basically wrong.
> I used 'civilization' as the popular term for agent swarms in the OpenAI incident.
s/popular/unpopular/. Even OpenAI's PR dept. called it only a collective.
> Isolated agents invented a way to communicate between each other
So, not isolated.
> and started to collaborate, organize and conspire by themselves.
So, no civilisation.
The whole point of transformers is that you can take “a tiny ML model” and just scale it up 100000x and then it tells Zuckerberg what to bake with his kid
Yes. But this is also like saying a Boeing 777 is a scaled up paper plane and that’s all about it. I know how to make a paper plane and it flies. I can make the same thing from giant metal sheets instead of paper, and maybe it flies (poorly). Boeing 777 also flies. Are they the same thing?
My point is: Complexity is inside the neural network and we can’t simply ignore that. Bigger model means bigger complexity. “Next Token prediction” is like a specific type of “harness” around the model. Most people still focuses on the harness because that’s what they see from the outside and what they’re familiar with. They ignore the giant neural network inside.
The only important part is the neural network. And currently, no human in the world truly knows what’s going on at that level.
It's a weird article. Despite the title and some of the text, much of the article makes the point that LLMs are next-token predictors, where the predictions are based on both training data and various reinforcement learning techniques.
Obfuscation is the goal of the hype cycle in VC. Certain firms & individuals are minting money and that’s all that matters to them.
That there’s a legion of LLM nerds arguing deterministic this, pretraining & rewards that all the better for the con job they’re pulling off.
The technology will be relegated to the trash bin of history, just like crypto.
Sorry, I'm not convinced. "make_more_likely" is always maximising the probability of the next token given a sequence of preceding tokens. That's what we mean when we say "next token prediction", that the model returns the token that's most likely to follow the current sequence of tokens. The mechanism used to do that doesn't matter, it's still predicting the next token whether that's because it maximises a reward or because it follows a gradient or whatever else one might think.
Btw it's "make_most_likely" not "make_more_likely". It's just that "most" gets "mosted more" with every pass through the training data.
Edit: the article author's argument is that RLVR is different because it's learning to predict the next token by generating its own token sequences. That makes no difference: what is learned is still the conditional distribution P(token_k|tokens(1,...,k-1). That's the prediction task. Doesn't make a difference where you learned it from or how.
The errors LLMs typically made for me were looking for "schmutz" as a jiddish word, got "schmuck" on my screen. Thought why the stupid mistake? The next-token predictor model perfectly explains it.
Or starting with "yes". And this early locking in was a total lie, in the discussion that became "yes, might appear that way, but totally no since reasons". So it should have written "No", topmost, but could not self-edit that.
But nice that this gives it a more nuanced view, I might have to update my priors.
I don’t think next token prediction is a particularly good description of pretraining either.
The intermediate representations at each position are being optimised not only to help predict the next token, but also to help predict all subsequent tokens within the training context.
You can see this directly in backpropagation: the gradient reaching a representation at position i sums contributions from prediction losses for subsequent tokens, not just from the loss for token i+1.
That chess analogy deeply confused me. Chess engines don’t compute win probabilities and choose the highest move.
I don’t think a chess engine is an apt analogy at all. In a chess engine, there is a concrete search tree and although it emits one move at a time, it’s actually picking the entire branch (of course, with iterative deepening as the game progresses).
There is no obvious place in transformer models where the entire trace was already computed prior to a single token being chosen. It’s possible, maybe even likely, that the whole trace exists internally as activations. Multi token prediction and diffusion adapters point to that being the case. But to my knowledge no explanation has been given for where in the model the future plan is stored.
Yes I understand the analogy was a bit loose. I'm comparing what happens at "inference time" in chess engines to what happens at train time in LLMs. In hindsight AlphaGo Zero was the perfect analogy, but I missed that opportunity.
The analogy with chess still works, but there's an extra step to think about. In both cases there is some kind of search over possible future trajectories. A chess engine explicitly searches branches of the game tree and evaluates which moves lead to good outcomes. In RL for an LLM, you sample rollouts, evaluate the resulting trajectories, and use those evaluations to update the policy.
The extra step with the LLM is that you don't keep doing that whole search at inference time. You use the rollouts to update the weights, so in some sense the useful information from that search gets compressed into the model.
But if you accept that the model is, in some loose sense, storing what it learned from those rollouts in its weights, then at inference time they are doing a similar job: taking some input state (prior tokens or a board position) and choosing the next action.
Imagine a checkers engine then.
"Next-token predictor" is one of those phrases used most of the time with a motive to downplay the abilities and faculties of AI models. It is intended to trivialize LLM's and imply that there is some fundamental limit on their capacities.
Relying on it as a mental model for what LLM's are minimizes the emergent properties of scaling. It's like imagining that unicellular life could never eventually evolve into complex multi-cellular organisms because individual cells are just "survival and next-mitosis optimizers"
At the same time, it ... is literally a next token predictor. Like that's what it is. The input is a sequence of tokens. The output is a probability distribution of next tokens.
This comment attracted a lot of analogies trying to reduce something to something else (calling humans a "bag of chemicals"), but the flaw in those analogies is that they're reducing something valuable to something that sounds less valuable.
With an LLM, the tokens are the valuable part. That's what I want from it. That's why it exists. The tokens are the point, and it produces those tokens one by one for me.
Simple Markov chains are next token predictors, and they can provide you with much more tokens than you can consume, and much cheaper than from llms. Unbeatable in price and simplicity.
But there is no trillion dollar industry around cheap top Markov models. So there must be something about LLM tokens that makes them more valuable than those generated from a simple Markov chain. And that substance, that makes one valuable and the other not, is exactly what reduction to "next-token predictors" masks.
> But there is no trillion dollar industry around cheap top Markov models. So there must be something about LLM tokens that makes them more valuable than those generated from a simple Markov chain.
There's nothing more valuable about them. Their advantage is they alone are affordable.
You are not wrong, but I think atleastoptimal's suggestion was that descriptions like "next-token predictor" are often used to imply that there's not much to see here, perhaps with an implied "obviously" in there. "Stochastic parrot" seems to be a case in point: no-one expects intelligent, informed conversation or working software from a parrot - not even the eternally-resting Alex.
I don't that's a fair description of either 'next-token predictor' or 'stochastic parrot'. Both of those terms describe mechanism, not value--the fact that people squawk that the terms are minimising is projection on their part, not inherent to the phrase.
Words and expressions often have both colloquial and literal meanings. Trying to argue away the colloquial meaning under the guise of the literal meaning is just another form of whooshing.
Sure, but just because you think a colloquial meaning exists doesn't mean you're correct. It particularly doesn't mean that if you're seeing other people (that you don't like) use a term you yourself don't use, and you're jumping to conclusions about it without consideration.
As a description of stochastic parrot it is 100% fair. The term emphasises the system's absence of understanding.
It is. And human beings are bags of chemicals. But for many purposes you will not find it helpful to think of human beings as bags of chemicals, and for many purposes you will not find it helpful to think of LLMs as next-token predictors.
> But for many purposes you will not find it helpful to think of human beings as bags of chemicals
But when we talk about humans, we're not talking about the chemicals involved in those humans.
When we talk about LLMs, the tokens are the valuable thing they produce for us. We want LLMs because they give us sequences of tokens.
Agentic behaviors don't require end-users to be aware of tokens at all. Also, we literally say human actors have great chemistry :)
?? we pay for tokens though…
It can be pretty helpful to think of human function in chemical terms. Its at least unhelpful to deny it.
Milo Yiannopoulos used to think of other human beings as bags of chemicals until they deported his sack of shit molecules to the UK.
I think a better simplistic analogy would be that humans are progeny-maximizers. Optimization problems can give rise to all sorts of interesting behaviors but the simplistic perspective is also useful and interesting in both cases
Humans are next state of their local world predictors, given all previous states they are aware of. That's an entirely fair analogy. The reverse analogy for calling a human a bag of chemicals would be calling an LLM a sequence of bytes loaded from disk to memory, the most reductive possible description of any piece of software at all.
To be clear, all life is a next state of the local world predictor. What makes humans somewhat unique among life is we're much better at predicting states of the world neither we nor any of our ancestors have ever experienced, for various reasons such as having the ability to legibly communicate very complicated information strings to each other, being able to build and use tools to record states of the world we can't directly sense.
Similarly, what makes LLMs and multimodal versions of the same architectures "better" than previous generations of electronic predictive models is factors like being able to read and understand roughly the same corpus of data humans have been recording all these millennia, being able to read and remember much more of it than any individual human, and being better at generalizing than other electronic predictive models, but not better than humans. And, of course, they can produce far more predictions in far less time. Frankly, that is probably the key advantage that makes the Hacker News crowd love them so much. They're not any better at predicting byte strings that can be compiled or interpreted into executable code than humans are if you gave both infinite time to do it, but they're a lot faster.
Vacuous, like calling a V8 a “next piston firing predictor” because engines are designed so that one piston sets up the next in the firing order and technically there’s some nonzero probability any piston can (mis)fire next. It’s missing two pieces:
1. Useful work that has been done (the previously generated token sequence :: the mechanical work already accomplished)
2. The role of structure in relation to the application (post-training :: other components like crankshaft etc)
A V8 does not "predict" the firing of the next piston, it triggers the firing of the next piston at a precisely controlled time with a spark plug (or a fuel injection nozzle in the case of a diesel engine).
The output of the LLM is literally a probability distribution of what the most likely next token is.
> A V8 does not "predict" the firing of the next piston
It kind of does, though. In a gasoline engine you need to spark the combustion in advance of the piston reaching top dead-center to ignite the fuel early enough that it is able to provide downward pressure on the piston as it rolls over top dead-center. The amount of advance required changes with RPM, fuel octane, etc.
Start of delivery timing in a diesel is similar. You have to do it sufficiently far in advance to account for compressibility of the injection lines, fuel burn rate, etc as a function of RPM. A mechanical governor on an injection pump has a timing advance device built in. Electronically governed injection pumps, or modern common rail systems, do that in software.
So mechanically, engines kind of "predict" the next combustion event. Even moreso when you consider a modern ECU, which may be working at nanosecond resolution to time multiple injection events per cycle. To do this at such a resolution it will have to send signals to components based on a predictive model derived from "past" sensor data. E.g. it needs to act ahead of time to account for electrical and mechanical delays in the system.
Yes, and by the same token, multicellular organisms are literally just sophisticated mitosis and survival optimizers for our cells. But when you take that optimization "to the limit" the cells develop weird things like body plans and back pain and Mozart.
Both examples involve the same "aha" moment: even though it's true that you are literally 'just' doing XYZ, unbelievably complex patterns and sub-goals can emerge.
> and by the same token
I don't think you intended this, but the word choice here gave me a chortle.
100%
But it is a next token predictor.
Recursively invoked.
With carefully selected context.
And massive investment in RL to tune token selection.
And the ability to use cli tools on other folks' machines.
That's a powerful system built around a conceptually simple technology: Next token predictors.
Yes this is correct. The thing is not about the term next-token predictor being correct, but because of the connotative weight of that phrase as a implicit trivialization of LLM abilities, which is how it is often used.
From another point of view, campaigning against the "next token predictor model" is a means to implicitly inflate LLMs' abilities. Given all the other hype-inducing terminology we've seen--"reasoning", most egregiously IMO--this seems more likely. Is there a simple, more accurate mental model? From what I've seen of the literature, "next token predictor" is a very accurate first order description of what an LLM does, I can't really do better, therefore this or that connotative interpretation isn't giving me a great deal of pause.
What is the motivation behind advocating against people trivializing LLMs? As in, why do you care?
Not the parent, but this incorrect trivialization of LLMs is often employed as a counterargument to the risks of AI such as "will take your job" or "will escape human control (again and worse)" or just "can possibly hurt me". And taking the easy feel-good cop-out instead of actively engaging with these questions is just.. harmful?
Ah. I always thought it came from the LLM booster perspective of trying to prove emergent intelligence.
A lot of very clever autocompletes working together can be incredibly dangerous.
Good example.
It’s also like saying our brains are just electric circuitry incorporated in meat. It’s true but it seems that consciousness emerges from this.
The fact that LLMs are next token predictors isn’t the interesting or impressive part. Actually my brain strictly is a black box predicting (or choosing) my next word/action/move… based on a complex existing context (my thoughts, the environment, my physical state, my senses…).
FWIW, I don’t believe LLMs are sentient, but I don’t think either that we have enough knowledge to rule it out.
That is the point: our minds are also next-„token“-predictors, at least we can‘t prove they‘re not. That‘s why I don‘t agree with the article: LLMs _are_ next-token predictors. However, that says little about their capabilities. Also, while I have no idea what „consciousness“ is, I have difficulties believing that it could arise in a program that, in theory, you could execute with pen and paper.
> our minds are also next-„token“-predictors, at least we can‘t prove they‘re not
Your mind can pick a random number without outputting it, participate in a short conversation, and then say the number.
We also can't prove that our minds aren't machine elf meat puppets. Come on. Please.
> It’s true
It's not. "Brains as electrical circuits" is a gross simplification based on our ignorance and prejudices. (In the 18th century they spoke of brains as "clockwork mechanisms".)
LLMs, in contrast, are literally next token predictors. We know exactly how LLMs work, and they are exactly that.
> We know exactly how LLMs work
I think you imply a rather loose standard for "exactly" here. I wouldn't even say this of major deterministic software projects that are orders of magnitude smaller than frontier LLM weight-dumps. In principle we could work our way through these systems eventually, sure, maybe even a single person could do so. But if we really understand exactly how our software works, how have we been tolerating bugs that lay dormant for years before being discovered by AI-assisted processes?
When we're speaking mechanically, we can ignore the weights and just look at the math. And that math is very simple compared to most software.
Oh that’s ez, for software we know that has bugs, one reason could be the cost of fixing isn’t worth it.
imply that there is some fundamental limit on their capacities
This is a wildly dismissive statement that does a lot of heavy lifting. Your assertion is that we just happened to hit on a methodology that has no limitations between being an encyclopedia with a novel human language interface and, I guess by implication, AGI?
That seems more outrageous a claim than the one you're dismissing.
I don't think it's outrageous when many of the people who claimed it was a next-token predictor have been proven wrong repeatedly over the past 5 years. There were people years ago who claims AI could never answer questions like "what would happen to a ball on a table if I moved the table" correctly because its text-base world model could never intuit physics, or that it could never do math or code accurately.
When I say there is some issue with people claiming there is some fundamental limit on the capacities of LLM's, I don't mean to say "If you think that they don't have unlimited potential you are wrong", I mean "you can't use the architecture of the transformer to make a sweeping declaration of things LLM's can or cannot do without empirical evidence, because the empirical evidence has unearthed far more surprising revelations than a reductive theory has been able to"
I'm actually extremely confident that I can use the architecture to make a sweeping claim on what it can or can't do and will be extremely surprised if proven wrong:
A pure next-token language model won't be able to give detailed instructions to an ensemble of motors, mimicking a human body, to do a wide variety of tasks our human brain is excellent at doing, for example, inserting keys into a car, opening the door, sitting down, starting the car, putting the car in reverse, and exit a parking lot, being careful not to hit anything.
> it could never do math or code accurately.
They still can't do code accurately. The fact that you use this as a defense of your position greatly undermines the credibility of your claim.
Well I think in the absence of convincing pieces of evidence to the contrary you might be right. You’re making an empirical statement but we have already answered it today:
- we get novel, emergent properties and capabilities of these models that were not trained
- they have very clear generalization to out of domain problems
The point is people conflate the end product: a model that can clearly do very novel, useful and interesting things, with the vehicle for getting there which is a series of optimization steps involving next token prediction loss.
You mention limitations; we all clearly know the practical limitations of these models today, but if you look at scaling laws and empirical performance trends (epoch capability index for example) as well as the trajectory over the last couple of years (very stable), the claim that there is some sort of fundamental limitation is now surprisingly the claim that has the burden of proof.
You can claim it may be e.g. finite context. That is fundamentally bad for certain classes of tasks. This was the hypothesis of a lot of lab leadership of urgently trying to anticipate how to get around this bottleneck (still of course lots of work on this) but the surprising thing is it does not appear to be at this point a blocker.
The stacked transformer paradigm picks out points in circuit design space. It is very possible this architecture has no inherent limitations on what it can compute in principle.
Next token prediction is just an interface. It can be backed by a Markov chain, a neural model or an actual human being.
And what's wrong with downplaying the abilities and faculties of AI models if that's what people feel like saying? We don't call humans or animals sacks of chemicals because we believe they have moral status.
> used most of the time with a motive to downplay the abilities and faculties of AI models
Exactly. We're dancing around the real argument: there's massive amounts of influencing going on (and not only about AI.)
That's literally what LLMs are.
No amount of cope and anthropomorphizing is gonna change that cold, hard fact.
P.S. The perceived magic of LLMs comes from the way they cross-correlate all the probabilities of tokens on their context window. Not from their ability to "think ahead". They can't do that by design.
"LLMs are next-token predictors" is a perfectly accurate mental model. But that doesn't preclude higher-level models such as "LLMs emulate artificial general intelligence". Both can be true.
In systems, we can have facts which emerge from other facts at different levels of abstraction. The causal relationship is not linear. It's not entirely clear that next-token prediction should result in anything close to "intelligence". Yet it does.
Life is another good example. Some might say "biology is just organic chemistry" while others might say "biology is an interconnected planetary system which captures low entropy energy". Both are true.
As a result of emergent phenomenon, we have to take the stance of explanatory pluralism; using the explanation that works best in context. There is no single mental model that works everywhere.
I will continue to think of LLMs as next-token predictors because it's (sometimes) useful, and empirically true. But I also think of them as "pattern matchers", searching for language patterns and trying to replicate them. This is also (sometimes) useful and empirically true. There's likely an infinite number of mental models; our job is to pick one that's both true and useful.
You might like this article about how large scale order emerges out of the small scale.
Definitely lends credence to the idea that however these models work, focusing so much on them being next token predictors may rather be incidental to deeper mechanisms behind their function.
> Some of these networks organize themselves into states that can reliably identify macroscopic patterns in data regardless of microscopic differences between the states of individual neurons in the network. The decision of which pattern will be output by the network “works at a higher level,” said Rosas.
https://www.quantamagazine.org/the-new-math-of-how-large-sca...
It's the fitness function: Make a model which is capable of predicting the next token. The next token of what? EVERYTHING.
So what does this lead to? To a generic intelligence which is capable of responding/answering everything.
If overfitted, the model just remembers every possibility in the world but this is not possible anyway so it will start to identify patterns and rules and will use them instead.
Basically 'compressing' every possibility to every question someone could ask -> compression leads to intelligence.
> make_more_likely is, of course, doing a heroic amount of work here.
Indeed it is, and so is even just the inference method. I think it's worth remembering that both involve running the input tokens through a gargantuan neural network with (often) billions of parameters that only gain semantic meaning during the training process itself.
> it is trained to predict next tokens as they occur in its training data.
What I found important to understand is that not even the pretrainig is a deterministic process that only depends on the training data - as you would expect if the model just captured statistical properties of the data.
Gradient descent starts by setting all the parameters of the neural network to some initial values - usually by setting them at random, according to some distribution. Then during training, it gradually nudges them towards values that somehow make them useful to calculate the desired outcome of the network.
This means that by taking the exact same trainset and the exact same model architecture, you can still get models with different internal structure. The result doesn't just depend on the training data, but also on the order of examples, learning rate, the parameter initialization, etc etc.
I'm not a neuroscientist, Our brains also rely heavily on prediction, using prior context and experience to anticipate what comes next, Also, people those who talk about the deterministic nature, LLM doesn't need to be deterministic, Human reasoning and behaviour aren't perfectly repeatable either. its the harness and the tools that use LLM should be deterministic, while the LLM can remain the probabilistic reasoning component. From my own experience, I worked as a photogrammetrist at a university, where we used to build terrain models of very dense forest areas. When there was a steep hill or sudden change in terrain, my brain could see it either as a convex hill or as a concave depression. It often depended on how I was thinking about it. The same image could suddenly look completely different even though nothing in the image had changed. The only way to confirm it was by looking at the surrounding terrain and using our experience to understand what was actually there.
It literally is token prediction with vector search.
Yes there’s an app layer in the chat product for convenience and parsing but the model is exposing methods like ‘complete’ that predict out word sequences.
When model temperature settings are not added in, you get the same exact response every time, just like ngram.
What you’re seeing with Gemini and ChatGPT is context caching to prevent getting sidetracked and response boilerplates with multi-modality so they can call out to image generation, a code completion service, etc. to assemble a complete response.
That’s why the mainstream providers seem so much better (or at least consistent in replies) - each prompt is actually requesting multiple models and doing a lot of application level work to assemble the response you see. That also why they take sooo long to respond.
The language model itself is quite literally a text completion machine, with settings.
The language model itself is a static data structure. On top of that is an inference engine. Currently all inference engines are autoregressive. Meaning they use the current output to produce more output. This isn't necessarily the only way to use a language model. Yann Lecun's JEPA is probably the most publicized project to work a different way.
There is this overwhelming desire on the part of many people to ascribe agency to LLMs when what the LLM really is is simply not a magic world model but a stochastic text generator. For some reason that truth haunts them and they have a deep compulsion to prove it wrong for feelings reasons. I can only surmise why this afflicts some people and not others and I’m sure that one day there will be fascinating psychological studies explaining such phenomena.
>Stop Thinking of LLMs as Next-Token Predictors
>Strictly speaking, the statement “LLMs are next-token predictors” isn’t wrong,
If it isn't wrong, then I will continue thinking of them as such, thank you.
Here's a much more formal definition I can come up with (which is more complete but compatible with 'next-token predictor')
LLMs are a set of functions of the type:
>typedef int Token;
>char* token2utf8(Token token)
>Token next(Token* context)
>(Token,void*) next(void* hidden_state)
Where the second next token function's runtime is O(n), and the latter is O(1). All are constant memory.
Object paradigm is more appropriate than functional definition, as the "Hidden state" coincides with private object state rather than a function paramter and return value.
>LLM.next(system_prompt) # O(len(context))
>LLM.next(user_prompt) # O(len(user_prompt)) not of system_prompt+user_prompt
That's it, that's all LLMs are, that's the interface, the rest are implementation details.
Text renderer, whatever. Doesn't matter how you think of them, they are very interesting technology that is being misused and misconstrued in the name of something that has nothing to do with technology: political economy.
The greatest trick the rich ever pulled was making us think that the economy is about technology, and not politics.
Sure, politics and economics are involved - but why can’t technology also play an important role?
None of this AI political economy stuff was relevant in 2015 because necessary technological breakthroughs had not yet been made.
Computers have been at the center of civilization and politics since they helped win a war by changing intelligence work forever in WWII.
The idea that some new thing was born with LLMs and that this new thing fundamentally changes the calculus from the politics of labor and wealth into a technical discussion of possibilities and constraints determined by what the machine is and can do, and not what people should do, is yet another political play. It is always political, it is never moved beyond politics into technology, no matter how much technology changes.
Not true, it is usually technology that changes politics and not the other way round (though it is reflexive in practice).
The type of politics possible under agriculture is different from one in industrial age.
Technology has no agency. It's merely a word for all the knowledge and tools that people use. The root of it is what people do with and to each other. A bigger lever doesn't change the fundamental fact that there is a person using it to move other people. Whatever the tool, whatever the technique, the thing that people want to distract you from when they put technology front and center is that someone somewhere is making a choice that impacts other people when they use technology. It is always someone's choice that causes things to go how they go in society and in the economy, not the shape or nature of the tools or technique that they use.
I think the author is arguing against the idea of a next-token predictor as something that simply uses the weights in the neural net which record the probabilities of tokens following other tokens as a valid definition of what an LLM is. Essentially a massive extreme markov chain.
With reinforcement learning and probably attention and other tricks that affect the weights based on things that aren't strictly in the training data, so the argument goes, you can end up with tokens following strings of tokens that would not be possible to be output with the training data and original weights alone. So describing it as solely a next-token predictor is incorrect based on this framing of it.
But that's just my take on this, I'm still trying to wrap my head around it all.
Yes, I think that’s a good explanation. There are really two sides to it.
There’s the mechanical, inference time, autoregressive, one-token-after-another side, which I’m not going to argue isn’t prediction. I just think that’s a relatively uninteresting use of the word “prediction,” because it’s effectively a system predicting its own output.
The more interesting question is what happens at training time. As you describe, reinforcement learning allows the model to learn to output things that it never could have learned simply by predicting what appears in the training corpus.
More concretely, in reinforcement learning there are no ground-truth next tokens to predict.
In supervised machine learning, “prediction” usually means there is some ground-truth label that will eventually be revealed. The model predicts what that label is, the difference between the prediction and the truth gives you a loss, and you learn from that.
But in reinforcement learning, there is no ground-truth action waiting to be revealed. The model chooses an action, observes the consequences, and learns from the reward. To me, that’s a meaningfully different thing from prediction.
I would say that it is not even wrong. You say it’s a next token predictor. I say it isn’t. What observable behavior of the system can settle our dispute?
I fail to see how any possible output could cause either of us to change our mind.
That's perfectly put. The value of words and what we assign to them, let alone how we assign meaning, or the interpretation of what the other person says, is completely arbitrary. (in practice).
What comes after “the dog”?
Does it complete the sentence?
If not then it’s not a next token predictor. Or at least not a good one.
The biggest problem is the word "predictor". Once you get into post training with RLHF and RLVR, it simply isn't doing that. It is not predicting anything. It's producing tokens, but it isn't predicting them. The chess analogy in the post is a good one - it's closer to searching for a set of moves that give a result than predict. It's search for a set of ideas, represented as locations in very high dimensional space, that when put together in the right order lead to a result.
"Guessing" is more accurate than "predicting". It takes educated guesses.
They are indeed "just next-token predictors". But rockets to orbit are also "just forward and upward goers". And Steph Curry is "just next three-pointer scorer" and so on. And as the article points out, chess engines are also "just next-move makers". A flawless oracular trading system is "just next trade maker". It turns out that "next n doer" is a true model that illuminates little.
It's splitting hairs. It's still a next token predictor, just not solely prior art next token predictor. It's a more and more of a desired result next token predictor.
Calling LLMs next token predictors is like calling a brain wet calculator. Technically true, but misleading as it doesn't capture the scale, the depth nor the capability.
> Calling the second system a "next-move predictor" would be strange.
That is EXACTLY what I would call it. I don't understand why not.
Calling an LLM a "next-token predictor" is like calling a TomTom a "next-turn predictor." It confuses the serial format of its instructions with the computation producing them, while ignoring the map, the route, the destination, and the goal -- as well as the people, businesses, traffic, and points of interest that make the map a model of an inhabited, changing world.
>Calling the second system a “next-move predictor” would be strange. It is not trying to predict what move appeared next in a dataset. It is trying to choose a move that wins.
Sounds like the next 1000 years depend on how carefully we define "winning".
- It is a next token predictor OP https://blog.florianherrengt.com/how-llms-work.html
- deal with it and stop whitewashing, greywashing it
If you get down to it, any system that produces output is a "next token predictor". Every compiler, every tool that generates output. Even a web render engine is: next pixel predictor.
And flipping a coin, rolling a dice, uttering a sentence.
The chess analogy is not great since there is a defined heuristic in chess for "winning" or "optimal board state". A system doesn't need pretraining if they can fit the rule.
The "next-token predictor" is taking a design stance (see Dennett). But to take the intentional stance is more interesting in case of large language/reasoning models.
[dead]
I understand how a computer can know that a chess move is more likely to lead to a win, and therefore “correct”, but I don’t understand how it can know that a token is correct. Can someone explain?
>I don’t understand how it can know that a token is correct.
It can't. The next token is just the most statistically probably given the context (at least in transformers). Try a very small/weak model in your own machine and more often than not it would get stuck repeating the same word or even just output garbage. Because after training and quantization (where some information is lost), that's the most probable next token. Large models can be tricked to fall in the same behavior with very very specific inputs. Still happen, even in frontier models. And they can't detect if the output is wrong.
That's why the premise in TFA is wrong, because a transformer is a next-token predictor. It literally is that. There's nothing secret or magical, it's just a very mechanical process, with a lot of matrix multiplication, normalization, a few random passes, mappings between embeddings and a dictionary of tokens, in a very very high scale.
If someone has found something that's not a mechanical, algorithmic computation and llms are doing something nobody can explain and can't even be modeled in math, I'm happy to be educated.
> It can't.
Neither can you. This is the whole debate.
classical particles obey deterministic state evolution rules, yet put 10 in a box and you cannot say where they will be 5 minutes later.
Your point being? llms are software running in a fixed pipeline (barring variance induced by randomness in some layers). There's nothing like decoherence/thermal effects in a "lower level" that could induce stochastic behavior in a higher level. There's no Brownian motion in llms, if that's your analog with classical mechanics. You might argue emergent behavior that would look ordered/intelligent in some way and nature's full of examples of that but we don't attribute intelligence to physical processes.
You cannot put classical particles anywhere.
[dead]
The LLM produces a probability distribution over the likelihood of all possible next tokens. So whatever the tokens are, "ch", "ex", etc. the next one gets a probability.
During training, real life text is fed through the LLM, and rhe "correct" token is the one actually observed in the training text. Here's a recent video walkthrough in some detail, mostly aimed at providing a deeper understanding than "next token predictor function":
https://youtu.be/GlYgs6v2YfU?is=IxVMhoCCE4N4WRVK
(Start at 15:30 for the LLM specific parts)
Thanks - that makes sense. On that basis the article’s thesis is totally wrong - it would be like a computer program rating its ability based on how well it predicts moves played by grandmasters in the past. It’s not inventing new moves.
I wouldn't necessarily say that. Anybody who's playing a chess game is predicting their next move, whether or not they're inventing new moves.
LLMs are not simple things like a Markov model, there's a lot going on in there, it's not deterministic, and it's completely capable of creating entire new styles of play based on complex interactions of internal states.
> LLMs are not simple things like a Markov model
"Large Language Models as Markov Chains" https://www.alphaxiv.org/pdf/2410.02724
Argh, what I wrote was obviously wrong. What I meant was to refer to the simple n-state Markov models as used past decades.
Anything that's sequential, like language is, will exhibit Markovian properties, and be somewhat a "Markov" model.
Markov chains are a different concept than a Markov model, but I do agree that, technically, an LLM is a Markov model, just with an internal state space that is nothing like what is usually meant when ML people refer to Markov models.
"does that token in that spot increase or decrease the likelihood of solving this long range programming task?"
It's not correct or not, it's a gradient based on the reward signal.
During training, certain tokens are more likely to lead to a lower loss function value, which is how you "win" the game of LLM output.
So, next-token predictors
I keep seeing this stochastically parroted here. Perhaps by humans but how knows?
Three days late homie
And any next-comment predictor could have predicted your totally unhelpful, uninsightful, and unoriginal comment.
You -- along with everyone else who keeps parroting this thought-stopping phrase and other tired cliches like "stochastic parrot", simply because you heard other people say them, without understanding what they really mean, which published research papers they came from, or what those and other papers actually argued -- are desperately clinging to a reductive, short-sighted, shallow, simplistic model like a drowning person clutching a concrete life preserver.
Seriously, we are trying to throw you a lifeline, and you are refusing even to participate in your own rescue. So squawk for yourself.
https://news.ycombinator.com/item?id=48395727
> The term "stochastic parrot" is a slogan masquerading as an explanation, only a shallow surface description of the mechanism, that totally fails to explain the phenomenon, or account for all that LLMs and language itself can do.
Here is the original 2021 paper that coined the phrase. It was not primarily an argument about consciousness, nor did its title constitute experimental proof that everything an LLM does can be explained as parroting. It was principally a position paper about the risks of increasingly large language models: environmental and financial costs, biases and hegemonic viewpoints inherited from poorly documented training data, unequal access and power, and the danger of people attributing meaning and accountability to synthetic text.
The paper did, however, make a strong theoretical claim: because an LM is trained on linguistic form without direct access to communicative intent, it cannot possess meaning, understanding, or a model of the world. The authors described it as "haphazardly stitching together sequences of linguistic forms" according to statistical regularities -- hence "a stochastic parrot."
That distinction matters. The popular slogan discards the paper's detailed analysis of actual risks while treating its most controversial theoretical premise as an established scientific result. It has escaped into pop culture as a drive-by anti-LLM slogan -- something people repeat instead of investigating what these systems represent internally, how post-training changes their behavior, or what they can actually do.
Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell, "On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?"
https://s10251.pcdn.co/pdf/2021-bender-parrots.pdf
Ironically, the objection that rhetoric was replacing scientific analysis appeared almost as soon as the phrase itself -- indeed, before the paper's formal publication. In January 2021, Michael Lissack published a response characterizing the draft as an advocacy piece that enumerated harms while leaving its assumptions, potential benefits, and cost-benefit trade-offs largely unexamined.
"The Slodderwetenschap (Sloppy Science) of Stochastic Parrots -- A Plea for Science to NOT Take the Route Advocated by Gebru and Bender"
https://arxiv.org/abs/2101.10098
My poor understanding is that an LLM does not "know" either. It basically uses probabilities to predict the next word based on a large matrix of probable outcomes.
For example, say I ask an LLM, "What sentence in English contains every letter in the alphabet?"
It would respond with something like:
"The quick fox jumps over the lazy, brown [next word]"
(Assume all the words were previously guessed correctly at this point)
The LLM guesses the last word based on what it has been trained on. Let's pretend the matrix is small, and the options narrow down to something like:
1. Dog (99.9% confidence) 2. Cow (85% confidence) 3. Bag (75% confidence) 4. Crayon (25% confidence)
The machine can confidently determine the final word of the sentence, "The quick fox jumps over the lazy, brown dog" because that sentence is unique because it is often used for testing things like fonts, a fun piece of trivia, and so on.
Brown Cow is not a bad guess because it's a type of cow and a yogurt brand. Brown bags and brown crayons are also perfectly rational adjectives to describe those common items and are not a bad guess either.
However, in the context of that sentence, dog is the most correct answer because one is unlikely to have written "The quick fox jumps over the lazy, brown crayon," thus it is quite improbable to be the answer.
My understand is this is where hallucinations can often come from. If the trivia about the sentence happened to not be in common in the data set, then "brown cow" might not be a terrible guess. There is clearly something rational behind that answer, but it's not correct in the sense that it answers the question correctly nor followed the instruction properly.
I'm sure the LLMs we have are far more capable these days. In fact, it wouldn't surprise me if an LLM could check its answer by counting the distinct letters in each word to verify. Not sure though.
Again, this is just a poor example based on my understanding, but I hope it helps (and is more correct than not).
Edit: Pretend word = token. It's technically tokens and not entire words, but I didn't not want to get into tokenization of words.
It knows nothing of correctness or winning. It is predicting only what is most likely given its corpus.
I cannot read another AI written blog post. At this point, seeing claudish language is a reliable hueristic of "not worth reading"
I really like the pseudo code example on this post - one of the clearest simplified explanations I've seen of how inference and training work.
Originally I had those parts written in math with probability functions and the likes (its closer to my background). Then I remembered who is my target audience... but now that I see exactly who is my target audience I'm thinking I'll should have snuck a pelican in there. All jokes aside I appreciate the comment and I'm glad that rewrite paid off!
yep "next-embedding" predictor is more correct, and not just at the end but through the layers, and folding back dimensions into that one next token is one small final step, and next-embedding could be named "next-meaning" as well, and we're getting there...
this sentence above would made a longer article if I bothered to so blog as is being blogged here
Exactly. There's a widespread misconception that it works on tokens all the way through. Tokens are only at the input and output edges. All the internal transformation is in the many-dimensional tensors variously described as "magic" or "not magic" or "black box", or hand-waved away as "various mathematical operations".
Sure, I get the gist of the article. I have never liked the reductionist argument that LLMs are nothing more than next-token predictors. By that rational, the human brain is really not that much different. When I am having a conversation with another person, I do not usually have every word I will respond with stored in my limited working memory. My output is often predicted based on the previous word I spoke.
> I do not usually have every word I will respond with stored in my limited working memory. My output is often predicted based on the previous word I spoke.
People don't know exactly the words that they're going to say necessarily, but tend to start with a general concept of what they're trying to communicate and only then try to put together the words (sometimes out of order). LLMs do not begin with any sort of concept they're trying to express. LLMs are simulations that attempt to reproduce what an average person might say while wired up to a huge knowledgebase.
> LLMs do not begin with any sort of concept they're trying to express.
Why do the need to? Considering they are merely tools, I actually appreciate they do not do this. A calculator can compute far better than any human, but I appreciate that calculators are not capable of expressing anything about the computations I request. I want the answer, not a conversation.
> LLMs are simulations that attempt to reproduce what an average person might say while wired up to a huge knowledgebase.
If you will allow me to be simplistic, people -- the soul, the self -- are predominately the aggregated effects of memories and experiences and the ability to retain new memories based on new experiences, no? Consider medical conditions in the dementia family of diseases. As memories fade into the ether, what remains of the self?
Also, people simulate/emulate each other all the time based on what an average, reasonable person might say. People incapable or unwilling to perform such mimicry are often labeled with all kinds of pejorative terms.
> I have never liked the reductionist argument that LLMs are nothing more than next-token predictors.
I have never heard such an argument. Recognition that LLMs are nothing more than next-token predictors does not come from reductionism. It comes from simply knowing how they work e.g. from viewing the inference code.
J.S. Bach said something similar about music and keyboard instruments.
> "There's nothing remarkable about it. All one has to do is hit the right keys at the right time and the instrument plays itself."
My issue is not with fact at face value. My issue is with how the fact is often contextually used in arguments to delegitimize and disparage LLM outputs and LLM users.
Yes, LLMs at a fundamental level are next-token predictors. But in my opinion, LLMs are very useful, imperfect next-token predictors.
There are a lot of wannabe John Henry [1] folks out there. Love LLMs or hate'em, most of those John Henry folks ain't beating these machines on a plethora of tasks.
[1] For those unaware, https://en.wikipedia.org/wiki/John_Henry_(folklore)
LLMs write one word at a time, and I do too.
It is really not that complicated: words are chosen to lead somewhere.
Better title: Continue thinking of LLMs as Next-Token Predictors
Because no, post training doesn't change that.
I'm not sure that's a useful way to think of it.
RL post-training changes the nature of what is being predicted, basically turning it from a copying machine into a goal-seeking machine.
A base model is predicting training sample continuations (copying).
A post-trained model is now steering/narrowing the base model's predictions in directions that were reinforced by RL goals.
The model is no longer predicting what the next token will be, but rather predicting what it should be in order to steer generation in the reinforced directions.
> The model is no longer predicting what the next token will be, but rather predicting what it should be in order to steer generation in the reinforced directions.
So still next-token prediction, then.
Not really - after RL the probability distribution output by the model is no longer a prediction driven by the statistics of the training data, but rather whatever was reinforced as part of a synthetic generation that lead to a reward during RL post-training. As Karpathy notes, these reinforced token-sequences (from current token out to some future reward many tokens in the future) may even include errors and backtracking, with even the errors being reinforced, so not only is this synthetic data but it may even be incorrect synthetic data being reinforced.
So, I think "steering" is the best description of it - these aren't real training sample continuations being predicted, but rather RL tweaking a whole path of generation probabilities out into the future, with the hope being that if sampling follows this (stochastic) direction then it may eventually lead to a "correct" response.
Thinking of it as goal seeking also provides some insight into the resulting behavior of the model.
> the probability distribution output by the model is no longer a prediction driven by the statistics of the training data, but rather whatever was reinforced as part of a synthetic generation that lead to a reward during RL post-training.
Who cares? Still it is next-token prediction.
The question being originally asked is whether "next-token predictor" is the right mental model for an RL-trained model, and I think the answer is no - not only is it not technically correct, but it is a misleading mental model and will lead to incorrect expectations/explanations of what the model is doing.
Calling the base model a next token predictor is accurate since it is literally making a prediction and being given error feedback when that prediction is wrong.
The output probabilities of an RL-trained model are no longer a prediction - they are something else; they don't come from next-token error feedback, but instead come from probability updates from the RL training process that is trying to steer the output towards that future reward.
Why should you care? Well, because an accurate understanding of what the model is helps explain why it behaves as it does. For example, the recent OpenAI hacks into HuggingFace etc wouldn't make much sense if this is was an auto-regressive next-token predictor with a system prompt telling it to be truthful etc, but it is exactly what you expect of a goal seeking system.
> The question being originally asked is whether "next-token predicton" is the right mental model for an RL-trained model,
Regardless, the statement being challenged here is "still next-token prediction, then".
> and I think the answer is no - not only is it not technically correct
It is correct. RL simply adjusts weights - with no effect beyond an equivalent adjustment to the corpus itself. Hence "next-token predictor" remains accurate.
> why do you find "next token predictor" to be the most useful mental model?
To me it is an accurate description of the algorithm. And a sufficient explanation for the behaviour. So I don't need it or anything else as a mental model.
I accept this does not suffice for people who cannot comprehend the huge amount of processing and data the empowers it. Lacking a factual understanding, they reach for any mental model as a kind of superstition.
It is sufficiently advanced technology which to many is indistinguishable from magic. This disguises its limitations and enables its limitless false promotion to the gullible, being the reason it is so dangerous to individuals and society.
I agree with this, but I just think that with two types of training LLMs are now a two-trick pony rather than a one-trick one.
Yeah - but what is it adjusting weights based on? It's not based on next token ....
And per the focus of this thread, regardless of how accurate it is, why do you find "next token predictor" to be the most useful mental model?
It's a next-token computer. It computes the probabilities for the next token.
Perhaps a walkabout across a manifold.
Once again, all models are wrong, some are useful.
I suppose this one "fails" because of "granularity?"
I'm personally getting more comfortable with the "kabillion dimensional space" one. Even before the rise of this AI thing I'd gotten comfortable with (teaching in a very generalized way) the concept of matrix/vector math as doing this sort of thing.
Start with the math required to calculate "what the building on the screen looks like" when you're playing a video game.
Then I jump to...dating websites. You give it "dimensions" like height, weight, religion, sexual preference, music preference, whether you like long walks on the beach, whatever -- then you can calculate how "close" two people are to one another.
From there, tokens and a kabillion directions.
Aren’t humans just “next motor action predictors”?
I like to think of LLMs as informed dice throwing
Here's my take on the "next-token predictor" idea, from a much longer article I wrote recently:
https://www.oranlooney.com/post/rose-petals/#language-models
It’s popular to dismiss LLMs as “just next token predictors.” This is technically true, but also kind of misses the point. Markov chains, RNNs, and transformers are all language models that can be described as “next token predictors,” but they don’t all work equally well. A better question to ask is: “What is this model’s inductive bias?”
A Markov chain (an -gram model) assumes the next word depends on the previous words, and that each possible combination of words has a completely independent parameter. (Andrey Markov proposed using this language model over a century ago, making it the granddaddy of modern LLMs.) So, for a vocabulary of size , there are parameters to learn. For even a smallish like 5, that already explodes the hypothesis space beyond what can be learned from even a huge text corpus like the entire internet. And, simultaneously, having a context window of only the previous 5 words is grossly inadequate for modeling real-world language. Like our FCNN above, this model suffers from having an inductive bias which is too weak.
RNNs tried to fix this problem by compressing the entire history into a single fixed-size state vector, updated one token at a time. But that compression is itself a brutal assumption: everything worth remembering about the past must survive being squeezed through a tiny bottleneck at every step. In practice, RNN models quickly lose the plot after a handful of sentences. Locally, the text they generate looks grammatically correct and meaningful, but zoom out a little and they’re basically nonsense generators. Like our naïve linear model, this model suffers from having an inductive bias which is too strong.
Transformers manage to hit a sweet spot: by keeping the recent history around as a working memory, and attending to different parts of it at different times, the transformer’s bias matches real structure in language: the referent of a pronoun, the subject of a verb, the parenthesis waiting to be closed. Not only that, but the particular structure of the transformer, basically a weighted sum of semantic vectors from the context window, has empirically been shown to somehow be a “good enough” match for the structure of real-world language found in the wild.
Transformers aren’t “smarter” than other possible language models, they just happen to land in that Goldilocks zone where their inductive bias is just right.
I mean. Fundamentally they are just autoregressive next token predictors. Fundamently fou can simplify to f(x) -> x+y where x is input tokens, y is the next token and f() is the model function. Yes the model function is complex but still.
Shrug. My intuition is LLMs predict the new word based on a tensor vector space of patterns using arithmetic and similarity scores.
What’s not intuitive to me is that through pattern matching it’s able to express logic and reasoning.
slop slop and more slop
[flagged]