This is a problem with long context models. To put it as simple and as bluntly as possible: just because they claim you can use 1M tokens in your context doesn't mean its true and you should do that.
Due to extreme quantization of models and the context's KV cache, and also just really shitty samplers provided to the user (hell, most are just getting rid of sampler knobs altogether), this problem will absolutely continue.
Want it to go away, almost like magic? Local inference. When its under your control, and no longer being forced to hold it wrong, all of the common LLM defects will go away.
> Want it to go away, almost like magic? Local inference. When its under your control, and no longer being forced to hold it wrong, all of the common LLM defects will go away.
This is just not true. Any local LLM you can host on consumer-accessible hardware has all of these defects, too. Adjusting the knobs doesn’t solve everything.
The closest you can get to frontier performance is Kimi K3, but you’re not hosting that unless your budget is on the order of a nice house in a good metro area.
I like my local LLMs as much as the next person and my office is currently uncomfortably warm from the amount of compute happening, but I would never agree that local LLMs solve all of the common LLM defects. This is peak wishful thinking.
In my experience, the local models and even the larger ones that we can’t run at home suffer more from long context degradation than the frontier models. You are exactly right that you need to manage context length, but even at fp16/bf16 the local models have a lower ceiling for usable context length in my experience.
This is a bit of a strawman. The parent comment wasn't suggesting that the inherent defects magically disappear nor was it suggesting that local inference today is sufficient for all tasks.
At the heart of it, self-hosting liberates your use cases from all the horribly opaque configuration, shadow prompting, etc. And local models are only getting better and more diverse every month.
"[...]" = "When its under your control, and [you're] no longer being forced to hold it wrong," ≈ "liberates your use cases from all the horribly opaque configuration, shadow prompting, etc."
and
"Almost like magic" is cliché hyperbole. It also means "not magic" in the same way that "almost like a dog" means "not a dog." It was immediately followed by the explicit clarification that was surgically omitted above, in absolute and conscious bad faith.
i omitted it because it doesnt change my point at all and it makes the quoted text smaller. not because of "absolute and conscious bad faith".
people can just read the original from the comment. its not like im quoting from a different website and trying to hide something. look 2 comments above mine and it is right there in full. anyone reading my comment would have seen the original comment already...
i have no idea why you took such offense to my comment, but you're reading way too much into it.
> "Almost like magic" is cliché hyperbole. It also means "not magic" in the same way that "almost like a dog" means "not a dog." It was immediately followed by the explicit clarification that was surgically omitted above, in absolute and conscious bad faith.
Incorrect. The omitted text doesn't change the relevant meaning of the quotation in the slightest. Why are you taking such offense?
> Want it to go away, almost like magic? Local inference. When its under your control, and no longer being forced to hold it wrong, all of the common LLM defects will go away.
> "Want it to go away, almost like magic? Local inference. [...] all of the common LLM defects will go away.
There's no difference between these two statements in the relevant context of the article, which is specifically "long policy documents do not reliably govern agents", or the comments above, which are specifically addressing how that issue is not solved by local agents.
I get your reading. My interpretation is that they’re claiming enshittification and that the local models don’t have that.
It’s an understandable view, but I’d be astonished by any local model processing very long context better than any frontier model (and now many racks are we talking).
Clearly it was meant that they will go away only if you use a custom harness, without the defects that now exist in all vendor-controlled harnesses, because of the non-aligned goals of the vendor and of the user.
While it was not spelled explicitly, it should be obvious that nothing can be improved if you run locally an LLM in the same way as the vendor. But you should not do that. It is easy to use better methods for sampling procedures and for prompt and context handling.
That speed is significantly lower that what can be attained with streaming from SSDs.
Not only in big desktops, but even in most recent mini-PCs, it is possible to read concurrently from one PCIe 5.0 SSDs and one PCIe 4.0 SSD, at a total sustained reading throughput of around 20 GB/s.
With an optimized inference implementation, it should be possible to overlap completely the computations with streaming weights from SSDs.
This should improve the inference speed to around at least 1 token per second, on a cheap computer, under $2000 even at the current super-inflated prices.
There are enough tasks where this would be useful. Obviously one should use for most tasks a fast small LLM and use the big one only when this actually saves time.
I'd rather not be thinking about tomorrow's meeting all night! Just trust "the robot has got this"... and scramble in the morning when the robot has failed. But at least I got a good night's sleep!
For values of "usable" that include "14.6 seconds/token". It's a cool accomplishment! And newer hardware would speed it up some. But I think I'd want something a bit faster before declaring it usable in practice.
At my org we've been building AI agents and one internal rule we have is to use at most 50% of the models context window with the recommendation to not go over 25% for large context window models.
Anytime I see a "1M Context Window", my brain always goes "Gotcha so a 250k usable window"
Every single analysis I've seen done by anyone has basically come to the same conclusion: assuming sane BF16 or Q8 model quant, and BF16 or Q8 KV context quant (ie, not intentionally screwing over the model), you get about 250k before it pukes.
Why 25% and not 12% or 40%? Is this an arbitrary vibestimate or you had some tests done that pointed you to 25%? I am genuinely interested in how others deal with context issues. Also I would think the usable context window is variable depending on the task, for example summarizing documents vs analyzing large, scattered and complex instructions.
250k because thats what the smaller models supported and theres better training data in that part of the window
there have been some papers suggesting that the useful context is even smaller, and stays fixed as you change the context window size.
as a more general case though, i think the possibilities for what youd need to include in training to have many different paths of text be well represented enough over the long window means the later tokens will almost always be a lot more random than early ones?
I'm not the above poster, but my experiences generally line up with theirs.
The first 20% of context is usually very solid. Agent stays on task, is responsive and focused on the original goal.
The last 20% of context is usually pretty bad. Agent tends to loop, repeat, drift away into sub-goals (or non-goals).
The middle 60% is a bit of a grab-bag. Sometimes it's fine, sometimes it's not. It varies by task, prompt, and luck.
The above 25% and 50% limits seem like reasonable and simple rules to keep agents in the productive zones of usage.
It's generally not worth doing extensive testing to try to find your exact threshold unless you're running exactly the same prompt at scale, on similar data.
If you are in that spot, I won't actually recommend langchain tooling (I've been somewhat frustrated with it) but I think the concepts are worth understanding.
I've used this for testing an agent that handles provisioning and permissions for developers with SaaS tooling (ex - if a developer asks, go provision an account [saml/oauth/etc] with the 3rd party, configure access, and let them know how to use it).
I'm about 6 months out of date at this point, so I won't speak for the latest models, but what was available early this year absolutely suffers from extended context lengths.
How do they go away with local models? It's a bug of all LLMs not just cloud vs local. As mentioned in another comment, they did test local models here too and those failed as well.
As parent implies, they're testing the wrong control mechanism. Why are you using policies instead of real controls over the weights and inference pipeline?
Well the answer is that VC-backed companies decided AI is not a domain expert tool for highly competent technical users, it's a magic oracle for the lowest common denominator. So you don't get any of the actually useful controls, just context engineering like that's fucking sane at all. It's like trying to program by navigating a git history. Not writing any new code, you don't have the ability to do that. No, you exclusively have the ability to move around a git history and cherry pick things. It's insulting that they want to charge money for this shit.
> just context engineering like that's fucking sane at all.
Meanwhile every model is at a battle to figure out how to stop you from jailbreaking it, which I assume takes a toll on your ability to prompt. Heck, if you read r/Claude and related subs during the early release of Fable there were a lot of complaints that it would downgrade you heavily.
Look, I enjoy local LLMs as much as anyone, but I think there is some motivated reasoning happening in this thread to try to make local LLMs sound like a utopia against those evil VCs.
And I think you didn't understand what I wrote, so let me reiterate: Of course policies are equally ineffective at strictly governing the behavior of local models. Why would that change? The actual problem is that somebody is attempting to misuse them for that in the first place. There are only two reasons for it:
- They have no other options
OR
- They have no idea what they're doing
Local models solve the first problem, and let you apply the appropriate tool by virtue of the control you have. The alternative does not.
In-context, your reply reads like you're asserting that anthropic's offerings give you all the same control as a local model, and that having this baseline of control over a computer program is "utopian". To me that sounds more like the second problem, and nothing can help you there.
> In-context, your reply reads like you're asserting that anthropic's offerings give you all the same control as a local model, and that having this baseline of control over a computer program is "utopian".
No, I’m talking about the idea up thread that having control over more settings removes all LLM defects with local models.
I have all the control I want over the local models they run. No combination of settings is going to make them outperform Anthropic’s models.
He didn't say that, and I didn't say that, because in fact neither of us were talking about just settings, so that's a pretty curious idea to be addressing.
> No combination of settings is going to make them outperform Anthropic’s models.
And even this is addressing something nobody was saying, and manages to come off as a marketing read.
The ultimate difference isn't necessarily down to remote vs local. You can run a "local" model on the cloud the same way you could on a local machine and have all the same control over it. You can modify the weights, inject behavior, stack on arbitrary LoRAs, use whatever runtime you want, etc.
The problem arises in the case of anthropic, openai, etc. where you can't do those things. I think they have LoRA services that they'll charge you for, but that's about the extent of it. Think of it like having a compiler and its source code, and the operating system it runs on vs simply having access to invoking it via a web service.
Changing the weights and controls does not alleviate the context limits causing LLMs, local or cloud, to forget over time. I was asking how local LLMs would amelioratile this but looks like there is no solution.
One of the biggest fixes I've seen is just getting rid of traditional sampling. But first, let me say something about quantization, just to get this out of the way.
Like, lets say you already did the sane thing, your model[1] is already either FP16 or Q8 (and quantized by a competent practitioner of the art, ex: unsloth or bartowski), and your KV context is already FP16 or Q8... which means you now are already ahead of the major companies.
Google, OpenAI, and Anthropic heavily compress both K and V to insane levels, which might not appear to be so bad on short prompts, but especially with thinking enabled, its sort of the equivalent of JPEGing a JPEG repeatedly. Every time the model thinks, and records its thoughts into the context, and then reads it back later to think more, it becomes further and further imprecise.
All models with heavy KV context compression go off the rails somewhere between a quarter and a half of a million tokens. Every. Single. One. Every team that releases a model that has a limit of a quarter of a million did this on purpose, and it was the smart thing to do.
Now, lets say you dip your toes into samplers; this includes stuff like temp, top k, top p, min p, etc. I won't describe what they do, there are already good ELI5 articles out there to help you with that. They are, however, the original samplers, and the only ones the big companies use. None of them use the newer samplers that massively outperform them.
You know what you get with most providers? Temp as a knob, and it only goes between like 0.0 and 1.5. What if you want higher? Nope! What if you want to tune the other knobs? Usually no, too (OpenAI seems to still offer it on their higher end API plans, but Google has eliminated all knobs, and Anthropic apparently removing everything but temp in the future). What if you want other samplers? Not allowed.
Even restricting yourself to normal samplers, what if I wanted temp of 100, and min_p of 0.9 and no other samplers? That produces sane results for creative writing tasks, yet I could never do this, even I was paying for some $200/mo plan at Anthropic/OpenAI/Google/etc.
All of these samplers also cause a sort of JPEGing a JPEG repeatedly sort of error, it is the third source of it (model quant and KV cache quant are the other two). Long context insanity is probably caused more by sampling error more than it does by model and KV quant error.
What other samplers are there? Llama.cpp impls dynatemp, mirostat, top-n-sigma, and some others.
The one that I think more people need to look at is top-n-sigma. Temp and top-n-sigma alone has produced results that, even on ridiculously complex and purposefully tricky prompts, let models that have 1M native context happily go to the very limit of it without any signs of tell-tale degradation.
Want to go have fun with your new found freedom? Get Qwen 3.6 27B in Q4_K_M from a reputable dealer, set llama.cpp to do Q4 KV (yep, after I just said don't do that), and then run it with `--samplers "temperature;top_n_sigma" --top-n-sigma 1.0 --temp 1000`, and then compare it to the normal recommended values of `--temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.00`.
You will find that a lot of problems suddenly go away: long context degradation vanishes (which Qwen 3.6 will still do inside it's 250k context, especially when both model and KV are below Q8), forgetting what you said or it said earlier goes away, losing the plot half way through goes away, overreliance on cliches (Qwen has its own form of Claudisms, but are more subtle and less grating) also goes away, hallucinations happen far less, and lazyness also goes away (Qwen 3.6 has no real lazyness defects, but Gemma 4 does).
I have tested those top-n sigma settings on code generation as well. It is better than stock, and better than the commercial offerings by any of the American providers, but I still don't think LLMs can replace human programmers: it still can't think nor reason.
But yeah, moving to more modern samplers has done more to unfuck LLMs IMO than anything else you can do.
[1]: Assuming it isn't a native 4 bit model of some kind; quantizing them correctly to work in a local inference engine without actually quantizing anything is a bit of a PITA. See unsloth's work on the QAT Gemma 4 releases.
The needle benchmarks show, that models extended context works for the part, that can be explained as: "I can access/adress that part of the input".
I have no idea, why in that context, the number of attention heads isn't mentioned. Models have a limited set of them and obviously, a model can focus at N max things at a time, which has to put an upper bound of long context support in some way. There's just more things to lose focus to (or, mismanage the limited attention heads resources - per token)
There are no "attention heads" or fixed number of things a model can pay attention to ... or at least not exactly.
After every prompt the model decides "I have a weight of 1 to distribute between every token in my context". If you have ten tokens, each gets a weight of 0.1 ...
... except it's not that simple, because the LLMs don't distribute that "attention budget" equally. If your prompt was "where is Paris", then any tokens in context it can associate with Paris will get a greater share. If the word Paris is in your context, it might get 0.3 or 0.4 weight, and close by tokens might get 0.2, while other (unrelated) tokens get 0.03 or something.
Now, add lots of context, and you start to see the problem: more context = greater distribution of the attention budget. Even if the prompt is about Paris, and the Paris tokens get higher weights, they are only getting (say) 0.002 ... while unrelated tokens are getting 0.001.
All LLM "answers" are just math, computation, based on the context and those weights. If it can't "focus it's attention" because it's distributed among too much context, it's far more likely to miss the relevant tokens (eg. the Paris ones) and give you an answer that ignores key parts of context.
But again, there's no fixed number of things it can pay attention to: it's a gradual degradation of the chance of it seeing key info it should, based on the amount of context.
EDIT: Ok, I guess there are an internal thing called "attention heads"; TIL. They work exactly as I described (essentially I described a single "head").
Still, what I just wrote remains true: whether you have 1 head or 96 or ... however many your particular model has ... the fundamental issue isn't the number of heads. It's not like the LLM can pay attention to exactly (say) 96 things at once: each one of those heads has all of the available context "competing" for attention as described, and then the heads themselves "compete" to produce the answer (although not through the same mechanism; it's more like the weighting between heads is baked-into the LLM).
At the end of the day, the problem isn't "there aren't enough heads to pay attention" and the LLM hits a fixed limit. The problem is each head is trying to weight the relevant stuff, and the more stuff there is to distribute weights through, the less likely it will get it right (and then when all the heads are combined to produce the final answer, it misses key context).
Just because something is called an "attention head" doesn't mean the terminology makes sense.
If I'm reading the article right, hungryhobbit is accurately describing a single attention head.
And... what having multiple attention heads means is that you do the "single attention head" thing several times, and average the results. There is no part of hungryhobbit's comment that changes if you make this adjustment.
In particular, he is clearly responding to this clause in the comment above:
>> the number of attention heads isn't mentioned. Models have a limited set of them and obviously, a model can focus at N max things at a time
Those are the "attention heads" that hungryhobbit is saying don't exist. You'll notice that this definition of "attention head" does not correspond to any part of an LLM; I fail to see the problem with saying that "attention heads", as defined by this clause, don't exist.
There quite literally are internals that are called attention heads in modern LLMs. Saying 'there are no attention heads' is objectively false, which definition you follow is irrelevant.
One of my early experiments last year with open source models and context size was with GPT-OSS 20B (the mxfp4, the "smart" 4-bit quantization). Even though it boasted a 128k context size it was bad at recall around 32k characters (didn't bother to implement the tokenizer for counting).
The recall text was a simple hash generated, filler text from a dictionary file and a request at the end to return only the hash from the beginning of the prompt. Past 32k characters the response contained hallucinations of characters or full hashes.
Just having large context size doesn't paint a full picture of capabilities, prompt adherence and other quality metrics.
This is conspiratorial speculation though. They did test many open weights models which spanned the full spectrum of performance: Nemotron 3 Ultra second-worst, GLM 5.2 top five https://arxiv.org/html/2607.25398v1
If you have a semi-recent MacBook with even 32GB, you can run 20GB models that are pretty damn smart, with room to spare for the rest of your toolchain.
If you’re reasonably connected to the code you’re writing and prompting the AI at the level of the code, not the level of the feature, you can get some fantastic results.
Sure, it’s not the completely automated dreamland that’s been sold, but it’s still a speed-up on par from going from assembler to a higher-level language, which is still immense.
And for effectively free, if you have a machine that would otherwise have been considered “developer-grade” for a lot of tasks anyway.
I tried this on my work M4 with 48 GB of memory and it's not as useful as I had hoped.
If you set a large context window you end up using all your memory. Prefill is slow. And the 10-20B models are somewhat poor at understanding intent and tend to think in circles, so if you set a small thinking budget to control context size the model ends up answering based on some half formed thoughts.
Or something like batch document summarization or image classification or whatever yeah it's great. But for "agentic" and coding workflows I think you would need a completely different harness setup and prompting style compared to what we are used to from the big proprietary models. I certainly have not yet figured out what that is.
What models have you found to work for which tasks? I find the local models very useful but not in a way that's replaced cloud models (yet, I remain hopeful).
A good setup will cost you the same as a decent house in a major metropolitan area. And then the price of a used car to upgrade your electrical to handle that kind of load, and get the cooling setup you need.
It seems like you mean to use your own agent (where you control the prompts), not local inference. A custom agent works just as well with cloud based models.
Long-context models do work as advertised... just not when combined with multi-turn conversation sessions.
It's my understanding that, for all current models having long-context capability, the "parts" of the model that allow them to do long-context processing, are mostly-untouched descendants of academic model architectures; where these academic models were designed around a very constrained use-case assumption: that anyone using more than a "normal" amount of context would specifically be trying to apply a reasonably-sized prompt to an unreasonably-sized block of plain-old embedded data, as a single-shot conversation. (Consider a task like e.g. "summarize this entire book." Context = [prompt] + [text of book].) This is the use-case all these architectures were refined and benchmarked around. And even today, this is still basically the only way these models' long-context processing capabilities work.
---
But I think it goes deeper than that. It's not just about how we've made long-context work so far. It's about what "attending to context" really means, and what that implies about what we'll ever expect a long-context model to be able to do.
I would describe long-context models as seeming to have two distinct types of "attention state" used when attending to the layer-1 input-vector / context window:
- They have some small amount (e.g. 8-32k) of regular "high-quality" attention state. This is likely inherited from the base model they are trained on top of.
- They also have a large amount of "low-quality" attention state. This is what the long-context training process has created.
The "high-quality attention state" isn't limited to referencing the first 8-32k of the context window, to be clear. The high-quality attention mechanism is just limited to "choosing" 8-32k of arbitrary data samples, each from its own arbitrary position in the context window, to form a bounded-sized vector upon which high-quality attention is then computed.
(You could think of this as the high-quality attention mechanism computing attention over the entire context, and then doing top-K sampling, i.e. zeroing out all but the K largest-valued elements of the resulting latent vector, where K=8-32k. That's not what it's actually doing—that'd require paying the very quadratic scaling cost that prevents us from just increasing the short-context window size—but it's a useful naive model for what it's doing.)
---
It's harder to directly explain what the low-quality attention mechanism does, so let me try an analogy.
(While no one can yet give a definitive accounting of the abstract computational processes the matrix-multiply-ops against a given LLM's weights encode, we can still speak of possibility-spaces, or analogies via information/computability theory on what sorts of abstract operations the LLM "must" be performing to achieve the results it achieves. This is that kind of thing.)
Think of an LLM's evolved context-window attendance mechanisms as being somewhat akin to the LLM being a regular CPU program, that has access to a 1. block of raw addressable memory (i.e. the context itself), and 2. a block of pointers into that memory (i.e. the learned first-layer attention over the context.)
Our LLM program's logic, through mutations during training, gradually evolves into a form where it could be seen as implicitly declaring and using a growing set of variables: some bound to static addresses in the block of raw memory (i.e. static context-window positions), and others bound to static addresses in the block of pointers, where the raw-memory-address-value of the pointer (and thus what data from the context-window it will read as when "dereferenced") is the result of a dynamic calculation over one or more other variables (i.e. an arbitrary weighting of the layer-1 input vector, where the resulting vector element is used as a selector, in layer 2, among other data that got sampled into "passthrough" vector-elements back in layer 1.)
Under this analogy, a model's "high-quality attention state" is the set of "named" pointer variables that it directly references in its logic.
As should be obvious, there can only be so many of these; you really fundamentally can't scale them up, without literally having "more logic" that defines and makes use of them (i.e. scaling up the size-in-weights of the model's layers.)
A model's "low-quality attention state", on the other hand, is the block of pointer memory itself, which can be as large as you like. Presuming the computational substrate permits the logic to compute on the addresses of memory within that pointer-memory block and then double-dereference them to get context data (and it does seem to!), your LLM's evolved logic can do "dynamic random-access reads" of arbitrary parts of the long context.
But, of course, the model won't be doing anything special with that data. It can attend to it, but not in a way that's unique to that data. It pulled in such data through generic logic that doesn't "know what it's looking at", after all.
---
IMHO, this analogy helps to make clear that while giving a model a raw "large-context capability" might help said model to attend to more data, it doesn't really help in making a model able to attend to a larger prompt.
Just by information theory, an instruction-following capability must imply that something analogous to logic keyed on those instructions gets embedded into the weights; and that logic, in order to activate and work, presumably requires its own state — i.e. requires that individual values from the context are getting attended into additional top-level named variables.
And while the short-context attendance mechanism does that, the long-context attendance mechanism inherently does not / cannot.
And this means that there is likely a practical upper limit (in terms of practical model-weight size + latent-vector size + etc) to "how much prompt" a model can follow during a single inference step. Regardless of how much data the model can "see" on a given inference step, its active "program logic" is only defined in terms of so many concurrent stateful computations.
---
Thus: trying to use a current long-context model's "low quality" attention to attend to a large prompt is just going to result in nonsense. None of these model families have ever been trained to follow a million sentences of rules simultaneously when answering a question. They're just going to attend with their inherited bounded "high quality" attention to your prompt as best as they're able, while forgetting literally everything about the prompt that doesn't fit in that bounded "high quality" window.
And anything that allows current long-context models to appear to interact well in multi-turn conversations with memory + KV caching + etc layers in operation — while also making use of its "low quality" attention to attend to large data — is some kind of hack, and will break like a hack.
this is cope (i mean, it happens a lot, but is not the cause of what is seem on the paper).
I noticed even with plenty of context, the model sometimes chose a path that "fools" the attention layers to bypass some rules. you can notice that the rules are not ignored, they are sidestepped. it's very predictable emergent behaviour after you see it happen. with more context, it actually gets worse to the point you will see some things obfuscated even.
Any model with a good score on this benchmark would have a good claim on superhuman abilities. Humans are pretty terrible at being thrown a long policy document and being expected to follow it
And while we shouldn't anthropomorphize these models too much, I wouldn't be surprised if many of the core reasons for failures are similar. Working memory is a limited resource; you can only focus on so many things at once; reasoning depth is limited; and many real-world policies are not actually meant to be implemented in the same way they are written and have insufficient specification of edge cases
With humans, we usually do the equivalent of RLHF, both via "training" with simulated cases, and via feedback while on the job. You would never hand a newbie a 124 page policy document and expect them to correctly apply it on the first task, or to do it reliably in the first month
The challenge with comparing these things to humans, is that humans learn. A newbie might not respect your organization’s set of policies on day one, but what about 3 months in? Or 3 years? Meanwhile there’s still no reasonable mechanism for automatically fine tuning LLMs or adjusting their harnesses to make them better at completing your organization’s objectives more successfully. They’re still overwhelmingly governed by the shared weights and harness policies found to be successful for the average case.
LoRAs or even full fine-tunes would be much cheaper than that, and with some investment in the right infra could be updated regularly. And at least LoRAs can be swapped in and out cheaply, making them usable in large-scale inference providers. But there seems to be limited appetite in offering this. Both Anthropic and OpenAI no longer offer fine tuning for current models
There was a recent-ish paper[0] from Sakana AI about baking facts from a document corpus into a LoRA adapter. Claimed near perfect recall on very large needle-in-haystack testing. Haven’t tried it myself though.
Is this because of catastrophic forgetting, or is that more or less solved?
Is it, alternatively, because loading your LoRA plus someone else's can't be done in the same batch, so inference is made harder or becomes impossible to do at scale?
It is about context window too so no matter how much money you put to train your model it can still have degraded performance in following instructions but it will be better to execute the instructions they can follow.
It is also about the harness and how that can help drive the model and pick and choose what to include or not in the context.
These LLMs do not understand the project. For them any next word is good as long as it was picked by the token predictor. It does not have any way to understand but only to do. probability distribution over their vocabulary and if that vocabulary is tainted and lost parts of the original context what is a good candidate there will not match the intention of the initial project.
"You would never hand a newbie a 124 page policy document and expect them to correctly apply it on the first task, or to do it reliably in the first month"
So the behavior policy should reside in the weights, not in some ever expanding KV cache context, like sticking it in the wrong orifice if the goal is to be reproductive.
Instead of putting the policy document in some tightly crammed RAM, shove it in the weights that already exist, online / post training.
It does make one wonder if there is some way to compute the change to the weights from the computed context (so that we can free it up) without essentially continued "pre" training RMAD after each communication round.
> Any model with a good score on this benchmark would have a good claim on superhuman abilities. Humans are pretty terrible at being thrown a long policy document and being expected to follow it
Hmmm, if AI has to grow in the workplace, it has to follow processes to the letter, yet claude code forgets by the second turn my "don't commit" prompt.
Of course claude code is a generic and crap harness over terrific models that can't fit bureaucratic processes, and is increasingly worse at doing so anyway since Opus 4.6 peak.
Exactly: The best way to deal with this for humans is to use procedural scripts for different tasks, referring to the relevant bits of the (declarative) documentation.
I would imagine that doing something similar (using agent skills for insurance) would work much better for AI.
No, it's more than that. We can't remember everything. I hired, say, two years ago; as part of my onboarding process I had to read a bunch of policy and procedure documents, which were full of stuff that I didn't understand because I wasn't really in the context yet. So at the time, to me, those documents were full of arbitrary text that I didn't really understand. Some of it was rules that I had to follow, but at the time I didn't understand why, so it's just arbitrary rules.
How many arbitrary rules can you memorize? Do you even remember them two years later? If you do, then we can get to your statement.
And your statement is true. Humans do not run every action through a memorized list of rules, to see if any of them block the action. We don't. We're not going to, either, no matter how badly the policy manual writers want us to.
Social effects are a huge part of being human. Quite often we just do what the people around us are doing. In general it's not a bad rule, but can quickly lead go mass casualty events when the crowd has no idea what's going on, or we're messing with dangerous things.
That's a great comparison, human vs ai on a wall of text.
The problem is that it doesn't fit the sales pitch of LLMs and agents - humanlike or better, repeatably, 24/7, for a fraction of the price, you just need to make sure that you give it all the rules.
Unfortunately we can't really have a meaningful conversation until the money vampires have left so we will need to reschedule this until after the bubble.
Yeah checks out with my anecdotal experience with Claude. It is pretty great at following instructions - for about 10 minutes, after which it seems to ignore things I told it before.
I have quite explicit and strong instructions (e.g. don't write massive comments, use existing functionality, etc.) in CLAUDE.md files which seem to get bypassed surprisingly quickly when doing real tasks. Yet if I tell it these things in a prompt during the task, it performs way better.
Result is I'm trying to resist adding more and more things to CLAUDE.md files which in some scenarios it does well but in other scenarios totally ignores and messes up.
This is not what the article is talking about. Its talking about policy documents not it forgetting something 5 prompts ago. In fact you adding things to CLAUDE.md is more what its talking about.
Conceptually the same thing though. Claude.md is sourced at the beginning of the session, so will be pretty far back in context, just like user prompts from the beginning of the session.
Claude is a next-token predictor, appending to a long text document. Prompts aren't an independent kind of thing from policy documents. It's all text in the backscroll.
No because the CLAUDE.md is included in the first prompt which is given priority in the context. On prompt 20, it will remember message 1 better than message 12.
I believe the correct static instructions are about getting it at the right starting point for whatever class of projects you're working on; not as a continued referencable or "HOWTO" of what it's doing. They're all just "grooming" the LLM for future instructions.
The coding harness is what's getting it to continually align to your current instructions.
Ok so what is the correct way to tell it "I don't care what is happening, you must uphold these rules at all times"? If it's not any configuration of .md files?
You need to let go of the idea that this is something LLM's can do. They can't. At best, they can bias towards rules conformance with more or less likelihood, but coverage and conformance both go down super-linearly as you accumulate more rules, more context, and more output in a session. That's simply the nature of how these tools work and you need to engineer your workflows around it if you want to use them.
If you absolutely need some rules enforced, you need to adopt some framework for validating those rules that then rejects, reprocesses, or repeats any session that fails to satisfy them. In the best case scenario, this is some traditional deterministic validator (like a linter, compiler, analyzer, exhaustive test suite, etc in coding) but if you need to process in stochastic space because its something rich and ambiguous like natural language itself, then you want to dispatch a swarm very narrow, task-focused subagents that each validate against a very constrained subset. (And prepare yourself to have those to fail sometimes too. LLM's are noisy and cannot deliver strict rule enforcement on their own.)
You need to make the rule concrete somehow. I call it a "control". So for example, instead of instructing it "always run tests before committing", you (or you have it) make a git commit hook that always runs the tests first and that refuses the commit if they don't pass.
In this case, it is an advisory control only, because the LLM can also unhook that hook. And of course, it could also just disable the failing test(s) with some bullshit reason. But it is far better than assuming it will comply every time.
Then there is the "hard control", which is the inviolable that the LLM cannot bypass.
You need to move as much as is technically possible to either hard or (failing that) advisory controls.
I've seen pretty much every model including Claude bypass commit hooks constantly if there's even a tiny bit of friction, instead of pausing and asking me to help fix why it can't run the hooks (usually something that needs to be done outside the sandbox like `npm ci`).
That's why it's critical to have these checks run in a context that the model can't bypass, such as via github actions that block PR merges. Generally works pretty well for stuff at work (where we have all this stuff set up as part of continuous integration checks), but it means that for personal stuff I have to set up quite a bit more infrastructure to ensure that models don't just skip running tests (which they LOVE to do).
>In this case, it is an advisory control only, because the LLM can also unhook that hook
you can in principle make a hook in the harness itself that will run an auditor prompt that checks it adhered to the policy, correct the model and also make it known in advance.
I think things like CI, linters, hooks, etc started as defense against humans eventually ignoring it (low blood sugar, under pressure, never cared in the first place, etc).
These seem like they would naturally extend to LLMs as well.
(I'm not pro-LLM on balance, but the solutions of today feel similar for humans and LLMs.)
It is almost exactly like the controls from the auditing world, except as applied to AI's instead of people. In fact, having worked at Deloitte in the past is what led me to make the connection immediately
in the plugins I'm using, it's basically _always_ adding information to the context. You're not going to do it manually; you can't go back in the context and add it because that'll break the cache. The way your programming harness works is by constantly reminding the LLM of the tools available.
A good programming harness is basically a stack. A good stack keeps building each layer. You _cannot_ pull things off the bottom of the stack because that's an expensive cache hit; but you can pull things off the top. So what your harness should be doing: <tools> <rules> <user content> onto each request _then_, when you get to the next request or result, pulling that out if there's some change.
So you can see it's the cache that's either exponentially growing or having to cache bust to keep it fresh.
Ask it to come back with a filled checklist and hand it over to a different agent with a fresh context window (three lines model). Or make it collapse the context and get back to the checklist.
Subagents whose only job is to review the actions of your other agents for rule compliance? It works reasonably well for me in complex workflows using Claude Code.
Can I ask how you set this up? Like is there some way to have that run automatically, similar to “auto mode” for approvals, or do you have to invoke it regularly?
Sure! I use an orchestrator main agent whose only job is to run subagents through the workflow process I've defined. Part of that workflow is to invoke a review subagent at particular points to check the spec, the implementation plan, and the code for rule conformance.
The review subagent has its own definition and gets invoked with a specific target, so the context is very focused on just rule enforcement and I don't have problems with it skipping rules.
The whole workflow is packaged up as a plugin, but you don't need that to get this approach to work. It should be sufficient to have the rules you want enforced written out somewhere, and to either kick off a focused review agent manually referencing them, or do something like I did and have it be a defined part of your workflow (depending on the automation level you want).
As a hobbyist, I find it difficult to figure out how to make Claude stick with some repeating things I want it to do after every major action, like re-evaluate the completeness of tests, update the documentation, etc. And CLAUDE.md/AGENTS.md definitely did NOT help there, sadly.
What I'm currently doing in a large refactor, is I created a super-run script; the super run script is devided into super-dev (Setup dev), super-test (run all tests), super-build (build artificats), super-e2e (test all artifacts), super-deploy (deploy finished).
Each super's sub functions should _fail hard_, and each script should be highly detailed; of course I'm not doing it myself, but in small increments of directed work, it can build up the necessary harness.
What I get is a CI that just starts with "run super-run.sh" and that gives it context, then each sub script provides context depending on if it succeeds or fails. If it fails, the agent is provided what it needs.
It's basically, you have to design the products of the AI to give itself the context. Another technique I'm testing out is a parallel set of files like <subject-module>.js, <subject-module>.test.js, <subject-module>.md which get pulled up if the Agent is looking for a file.
If the harness runs on your localhost, but the inference doesn't, it usually means it's calling some API. Whether you want to break TOS of your provider like that or simply buy tokens from our friends from UTC+8 timezone is an open question.
This is the way. Making your own agent to have a sticky memory context that is prepended to every execution is necessary to ensure each task is bounded by those precepts.
The trick I'm doing -- the model is given a tool that runs a prompt in the current thread to consolidate it's working memory and identity (it has a memory tool bound to the agent persona). When the prompt ends, the parts of memory that are marked as identity are merged together into a new system prompt, then the context restarts with only system prompt and this tool call surviving. Then it just keeps going.
<project_prompts> //Duplicated with new user messsage
<llm_response>
It wouldn't bust cache, it would just make your input prompts slightly larger. Technically inefficient as you're duplicating the same rules over and over but I imagine for a smallish checklist/principles that it is tremendously more efficient than a cache break every message
so first, we treat context like a stack. Things are popped onto the stack; if we try to remove anything but the top of the stack, we'll break the cache.
So a good harness should be popping tool prompts, user hints after every return from the top so the context of tools and user hints never repeats and are allways prefixed to the current user prompt.
I've never inspected any harness, but opencode with the dynamic context compaction plugin appears to do this well.
By the time it does 10 turns of tool calls it forgets about the nagging again. It can and does ignore direct instruction from the user prompt too, sometimes repeatedly
I’ve had a lot of success using the root Claude.md for a handful of high level application wide rules and directions (I keep it pretty small), module specific claude.md in subfolders alongside the code with more specific rules and direction, and a custom rules backed /code-review skill that enforces it all and catches anything that was missed during implementation.
Most people don't understand that 'agentic AI' is a completely synthetic, force fed capability by extensive Reinforcement Learning on synthetic domain specific 'agentic' datasets on post training.
If the LLM wasn't post-trained to adhere to specific handbook, it just won't work. If the LLM wasn't trained on an use case the lab decided was worth making a synthetic agentic dataset, it won't work as well as you want.
There's a reason the main agentic task LLMs excel at are coding tasks, it's the way of working of the creators, and they understand intimately the flow and can train for it.
I believe the true way will be able to easily fine tune models on your agentic use cases, but it would require a big company to compile a huge dataset on it's way of working and I don't think anyone wants to be the first.
In terms of long context, accurate attention retrieval from early tokens is just impossible, given the expansion of RoPE encoding for the positions, or in case of Kimi that don't use it anymore, as well as deepseek, early context is heavily compressed you lose accurate information.
If people spent more time studying about AI and how it works, they would realize that the default should be to one shot prompt your task with a big, cached system prmopt, with an user prompt that is just dynamic data, specified to the cheapest model that can do the job.
Unless you really can't do this given your problem, you should try to make a graph of well defined, step by step oneshot prompts, and THEN if your problem still can't be solved with that, then you start leveraging agents.
Despite this giving better results, and being more cost efficient, is evidently too much work then just letting the AI do all the work.
> In terms of long context, accurate attention retrieval from early tokens is just impossible, given the expansion of RoPE encoding for the positions, or in case of Kimi that don't use it anymore, as well as deepseek, early context is heavily compressed you lose accurate information.
"Every gambler knows the secret to survivin' is knowing what to throw away, and knowing what to keep." - Kenny Rogers
Humans have limited context, just like AI. The difference is that humans - at least some of the time - can figure out which pieces are more likely to be important, and therefore prioritize keeping those in the context.
Most people jump straight to agents when what they actually need is a graph.
Example: a mining company receives free-text reports from field geologists. You could have:
Geologist report -> LLM call extracts minerals we are looking for (you inject a db query result on the user prompt), locations, assay mentions and risks into structured fields -> LLM call classifies evidence into positive indicators, negative indicators and unknowns -> LLM call estimates deposit potential and confidence -> database lookups inject regional ore demand, nearby deposits, infrastructure and historical yield data -> LLM call combines geological evidence with business context -> LLM call generates an investment recommendation and rationale.
That's what I mean by graph. Every step is a separate LLM call with a well-defined responsibility, consuming the output of the previous node. Each node can be tested, benchmarked, retrained, replaced, or monitored independently.
Why would you use an agent here? You can cache every single system prompt on each call making your total token output much cheaper than having a full 'output' only token generation workflow which is what happens with agents.
There is nothing to discover. The workflow is already known. The company already knows how geologists evaluate prospects. The company already knows what data sources matter. The company already knows what the final output should look like.
You don't want the model deciding which tools to call, which reasoning path to take, or which pieces of information are important every single run. You want the exact same process applied to every report so results are consistent, measurable, auditable and debuggable.
My default is:
One-shot prompt -> if not enough -> graph of LLM calls -> if not enough -> agent.
A surprising amount of enterprise AI is really just:
Unstructured input -> extraction -> classification -> enrichment from databases -> decision support.
Not:
Unstructured input -> autonomous agent spends 20 steps deciding what to do next.
Agents make sense when the workflow itself is unknown.
If the workflow is already understood, a graph is usually cheaper, more reliable, easier to evaluate, easier to debug, and less dependent on whatever synthetic "agentic" behaviors happened to get reinforced during post-training. I am sure people default to agents mostly because it's less engineering work than explicitly modeling the process.
Thanks, that helped. I get it. Yea, that’s the harness executing the workflow with the LLM being called at the right time. That ensures the process is consistent no matter what, in contrast to the agent calling out to tools and possibly doing different things every time. Basically, standard code being in control rather than the LLM being in control. Fully agree with this model. We should use deterministic code when we want the same process or algorithm every time and choose LLM callouts when we want “fuzzy” processing that is not as deterministic.
This is a great point, and one that has been blowing my mind for a while. The public think of AI as this black box thing approaching human intelligence. They don't know that when you look under the hood of a lot of ai products you see a string of prompts that anyone could put together themselves. The companies are just putting them together in a workflow.
The general public doesn't understand how these things work, at all, and the marketing makes it sound like it is magical.
That's why this paper is important. They gave it a scenario very similar to what would happen if an AI-frenzied executive suddenly mandated that AI be used to do basic company tasks - a scenario that is playing out all over the place right now.
I jump straight to graphs (unless it truly cannot be solved with a graph), but then the stakeholders get upset that I didn't use agents. Doesn't fit their marketing plan, I suppose. I don't mind dealing with that, but I can see why most jump to agents in order to avoid the human conflict.
Sure. You could easily have the model choose a workflow/graph for execution and then call a tool that implements the execution engine for that. The tool then makes calls for model inferencing at various points in the processing.
I called it a graph because really its a DAG (Directed Acyclic Graph) , in my example I kept it simple so there was only one input to every node, but in more advanced use cases you might have a synthesis node that takes in 2 or more inputs making it more akin to an execution graph in Tensorflow for example
There was an article a few years ago called "Lost in the Middle: How Language Models Use Long Contexts" https://arxiv.org/abs/2307.03172
From my experience this holds true to this day. It was one of my core observations for similarity to the limitations of human working memory on "Engineering for Bounded Cognition"
Long policy documents are also a problem for human agents. Without special training no one will retain 180 pages HR employee handbook, fire codes, OSHA safety rules, FCC regulations, the US legal code.
If the stakes are high, e.g, proceeding in ignorance could lead to prison time, people will favor inaction, even if the policy technically permits a corner case. If the stakes are low, people will completely override policy for the path of least resistance.
This is real. I was very angry that AI kept breaking the rules I wrote, so I asked Claude to crawl its own history logs. And then I found: every time after it breaks a rule, the chance of breaking rules goes up.
I feel this is like few-shot in reverse. Few-shot is supposed to be good examples AI should follow. But when it breaks a rule, we correct it, it keeps breaking, and this whole thing actually raises the probability of more violations.
I wanted to know if there is any difference between writing rules in the prompt, writing them in CLAUDE.md, or not writing at all. So I did some short tests before. I asked Claude to open brand new sessions, test different topics with the rules I want to apply. The result turns out to be: in a fresh session, no matter the rule is in a prompt or in CLAUDE.md, models (Opus 4.8, 5 or Fable) all follow it fine, across models. Even Opus 4.8, the one always violates rules in our conversations, does it well.
I suspect it's the long context that breaks rules. But simulating a long conversation experiment is kind of hard, I still haven't found a good way to test it. So seeing this paper now, it completely answers the question I was stuck on these few days.
Besides, something caught my eyes in this paper: sometimes the model does run the check by the rules, and it really finds the violation, but its narrative still insists on its original wrong output.
My current approach is same as everyone here: use a separate hook or post-check to fix things. Because if you let the model fix it during generation, its narrative or main generation part sometimes just rejects the rule error it found.
>. But when it breaks a rule, we correct it, it keeps breaking, and this whole thing actually raises the probability of more violations.
In Pre-LLM days the 'nearest unblocked neighborhood' problem, where patching out one issue just immediately runs into another issue, or a different path back to the same issue. Since the models can learn new long time behaviors it's difficult to change the behavior without changing the context quite a bit.
For Claude, I used a UserPromptSubmit hook running inject_rules.py which reads RULES.md from the disk and prepends the whole thing to every prompt. That helps the rules to stop fading as context fills because it is reinforced every prompt.
Sure, it uses tokens slightly faster in the prompt, but I find it reduces overall token use, you can use it with pro, but of course, nothing works 100% of the time, but it's better. Emptying the memory helps too to avoid Claude making up stuff that messes with how I want it to act.
The general gist of inject_rules.py is :
RULES_PATH points at RULES.md
reads it with encoding='utf-8-sig' so the BOM is stripped
wraps it in a JSON object — hookSpecificOutput.hookEventName = "UserPromptSubmit", additionalContext = a preamble plus the full rules text
the preamble is the line you see above the rules: rules are in force for this turn, run rule 33's five tests before raising anything unasked
prints that JSON to stdout, which is how Claude Code takes it in
on OSError it returns 0 silently — if RULES.md is missing or unreadable, nothing is injected and the turn proceeds with no rules
Just the first thing I thought of to reinforce the rules. I don't use all my tokens anyway (now), so for me, there's no reason to introduce a point of failure, just reinforce every time.
This article to me also implies that there are some potential issues with large Spec based development flows, which I haven't been able to pin down lately.
Specifically, having agent implementation drift from the Spec.
I have run into the same problem, and started calling the phenomenon vision drift.
I noticed that the issue tracker I built was well suited to address this problem, as it handles board time-travel. So a simple command like ":replay 4h" gives you a clear overview of how the workflow has evolved over time, and you can checkout previous states at will.
Drift is huge between any large spec and agent implementations.
I've done a ton of testing and the model doesn't matter, fable or sol still miss a ton of detail and drift.
I'm building http://engine.build which closes the gap and makes sure the implementation matches the spec.
It's not the same as the satisfaction you get when solving complex problems with code yourself but writing clear specs and thinking through the problem is still very satisfying to me.
I noticed this behaviour a few months back, I think I was using Sonnet 4.6 at the time... I have strict rules about comments in the codebase, this all for personal projects, and the reason I restrict comments is to keep the token count low.
At some point between the model i was using and the previous version of it, Claude started inserting massive comments with references to tickets and other tasks. All this while having specific directives on the CLAUDE.md
Since then I resorted to developing my crapware as if I was the floor manager of a vehicle assembly line, and I have a few highly-specialized sub-agents running errands around the main session, but only ever taking care of a single concern. The main session builds with the knowledge contained in things like CLAUDE.md but the sub agents make sure things like the no/low-comments directives are either enforced, or factored into the final product.
What really helped me was to run any .md/prompts through an LLM to find contradictions, duplicates or ambiguities, repeatedly. That led to agents much better following instructions.
Codex has been pushing things to my main branch all week despite me repeatedly telling it not to and adding to my AGENTS.md very clear instructions for creating feature branches and putting up a PR. It keeps doing it in spite of all that.
I'm probably going to need to enable branch protection on my personal projects... What a pain.
The rules in main prompt and context is not working well in any times with LLM. The only way is to test results and let agent validate its own result against the rules.
It will be interesting to see such benchmark. How the different models behave.
If I want to be sure I'm using my own deterministic rules checkers as a tool, that has to AI run and update the result accordingly.
Hi I just read your methodology and I have a quick question about how the handbook are parsed and feed into the context window? The article mentioned that each handbook contains roughly 8K to 79K tokens of extracted text, and did the harness system use grep or search tools to find relevant chunks and feeds to the context, or did it just feeds all the pdf output to the model? There might by distribution bias between real world tasks e.g. Agents grep keywords from docs and only use the relevant chunks. So all the models of the overall pass@1 is relatively low compared to real world scenarios, that might not be the same precision that user experience when they actually handle the daily task? How did the benchmark bridge the gap?
LLMs don't really read documents like parsers do, from what I've observed, they behave like they first skim the document and find the section that they think are relevant to their task, they don't really try to read everything and hold it inside their context.
The solution to that is pretty simple: get to the point, list all the requirements that they must do complete for that task, constraints on what they must not do, and optional recommendations for them to follow. Leave things that they can figure out on their own out of these documents.
Not always. If an instruction is short and vague, then LLMs tend to fill in the blanks by guessing and the results are unpredictable. Semantic density is the key, prompts/instructions should be actionable and unambiguous over being terse and vague.
“It can scarcely be denied that the supreme goal of all theory is to make the irreducible basic elements as simple and as few as possible without having to surrender the adequate representation of a single datum of experience.”
For my own (rather idiosyncratic) harness I've been experimenting [0] with "compiling" long markdown specifications into small executable logic programs. It's too early to tell for sure, but I believe that this approach does have its merits when you want some guarantees about how your agents behave for longer tasks.
Interesting idea. I’ve been noodling about something similar myself for a few months, but I haven’t moved forward with testing it. What sort of outcomes are you seeing with it? IMO, we’re never going to get to AGI without fusing “soft” AI decision making with “hard” logic and symbolic algorithmic reasoning. Humans don’t realize this most of the time, but we routinely use them all.
Sorry, did not notice your comment until just now.
So far I am observing two things:
1. For smaller models, performance on Benchmarks such as DeepPlanning does increase significantly.
2. Context hygiene for sub agents becomes much simpler, since that can be expressed relatively concise and the mechanics are handled by the runtime automatically.
Still looking for a good test cases to study possible advantages, but running reliable benchmarks does take time and money...
Are you the Deepclause author? I tried it yesterday and my first reaction was that it was slow. Perhaps I’m doing something wrong, however. Running against the same model in Pi was lightspeed in comparison. Second reaction is the prompt editor needs to handle more than a single line of text and it needs Emacs editing key bindings like Pi. I’m happy to do some testing on it and provide further feedback. What’s the best way to submit that? GitHub issues?
Yes, I am the author and thanks so much for trying! Please do submit a github issue. My first suspicion about the speed is that maybe an inner loop is taking too many turns until the model finally realizes that a task is finished (so that in turn the runtime knows whether the predicate failed or not and can continue execution accordingly). Happy to take a closer look!
The point about multiline prompts is very valid obviously, that's on the todo list.
This is what spec driven development tries to solve.
Multiple rounds of generating small contacts documents that grow from the original idea , trying to keep each slice small enough to process for a human to approve/disprove .
Eventually it leads to a long list of tasks grouped by functionality. You start a new context and the orchestrator agent dispatches tasks to sub agents with a limited amount of information provided to each sub agent.
Also should have adversarial review and approval gates with other agents and roles.
I have not seen any real life examples of successful application of SDD. I heard a lot of marketing BS, but nothing real. My own experiments with SDD produced very disappointing results even on very small projects.
I've had good results from SDD, and so have some of my teammates. The guidance I've heard over and over again is: the more you define, the better it does. I find that breaking things out into individual steps is helpful. However, your mileage may vary. Things like Gastown and GetShitDone, I have not found to be particularly useful.
> Failures follow consistent patterns: agents let a plausible in-environment request override the standing policy
I prefer this behavior in a lot of situations. A plausible request can also be a genuine one.
Alignment at the grain of each tool response is way more effective than a static system prompt. I try to keep mine under 500 tokens. Why bother fighting recency bias? It's pretty much what you want most of the time.
Opus 4.8 (max thinking) scored highest and Grok 4.3 lowest
It's hard to understand what's going on with Grok. It's like it has capabilities in a theoretical sense but maybe the training is so focused on being in x.com/grok.com with the web search tool enabled for "is this true?11" type queries that with any API type usage with document workflow instructions, tool use, code gen etc it completely falls over
When "performance" breaks down over long lengths, one could attribute it to a lapse in attention, but one could equally suspect inconsistent instructions.
The fewer instructions and conditions that need to be simultaneously met the easier it is to comply, but with more and more instructions one is bound to introduce internal inconsistencies within the instructions.
Why would anyone think that models optimized for efficient context management, giving much more weight to a short sliding window, would attend to distant, heavily diluted tokens?
Plus the model's capacity to take more context into account and actually integrate it to the output is simply limited by the number of activated parameters. If you give it a playbook, you are forcing to choose it between attending to the playbook and the task at hand.
If you want to force it to work step-by-step, you need to present the steps one-by-one. Ideally with rules for the current step at hand and maybe relevant input again, depending on overall task size.
Why did you think models love to re-read files before editing them? It increases recall quality and thus edit precision and thus benchmarks.
To run at decent speed, all models try hard to use only most likely relevant part of the context and most likely relevant weights (MoE) to predict the next token. Doing the math in full is unfeasible.
Control vectors for the win? It feels like the way to fix this is to pull out a control vector immediately after processing the handbook and use that to steer later inference. That should stop the drift over long distances, but it doesn't guard against the handbook itself being too big.
Evoking Gödel via GPT, with the requisite em-dash: "No sufficiently expressive formal governor can be simultaneously consistent, complete, and able to prove every relevant proposition—including all propositions about itself."
But "Read AGENTS.md, including the middle" sure helps.
Dealing with agents/LLMs based on "instructions & guidelines" has taught me - nothing (un)reliably governs agents other than agents themselves or (rather i.e.) their motherships (assuming they can and they intend to). Or if you add ton of local tooling.
Having only dipped my toes in generative ai recently I'm surprised how small even a 1mio window is. A semi serious project can take several session in one sitting. Especially as degradation sets in waay before the window is full.
Dont try and handle the entire project in context.
Use a well structured filesystem layout for your code with a few lines in an AGENTS.md describing the layout and core architectural requirements (no more than that, as per the article!).
Then work on small-medium tasks at a time with a fresh context.
At the end of your task, ask the agent if there are any key points about the project layout or architecture it would want to add to memory - audit those manually and amend your AGENTS.md accordingly.
If your code is well structured and you keep your tasks localised, you can get away with seemingly minuscule context windows.
It's also worth noting that high effort models love to slurp up whatever context they can. You almost never need/want high effort for non cross-cutting tasks.
Of course - its only a few years old :) But I think the biggest problem is that no-one directly tells you how to use the technology effectively. Mostly because no-one really knows, hence the reason for all the competing harnesses, agents, etc - the models are all just intelligence on top of that tooling.
I don't understand why a model has to follow the instructions. Don't get surprised when it shows its true color. Plus, do users (not the researchers) really check whether the response follows the instructions?
Models don't have to follow instructions, but during RLHF that is one of the things they are scored on so a premise of the idea is part of the model, but it's also balanced on accomplishing the end goal. They model may determine, correctly or incorrectly, that your rules suck and do what it thinks is best.
I got pumped seeing the OKF format from Google (which is just a standardization of wiki pattern) but quickly realized it could not yet combine many subtle concepts together in an efficient way.
LLMs are not people. Employee handbooks, like all other social rules, are not self-executing; they depend on the fact that human beings are innately social animals.
yep all the advice about context engineering, harness whatever is so silly. ai doesnt give a flying fuck about some IMPORTANT instruction in your claude.md.
It does what its has been trained to do. So find out what its trained to do and just use it to do that. This is not general intelligence.
Yes. And all the “context engineering”, document/decision frameworks etc just make it way harder to work with the ai. You put in a lot of work just for an illusion of control.
simonw has been correctly asserting that there is no deterministic way to prevent hallucinations. I’d argue that this obstinance could be the seed of a strong argument in favor of their capacity to eventually coalesce some form of consciousness
This is why the future is in deterministic static analysis and policy-as-code frameworks.
Policy-in-English? Model implicitly complains that it's TL-DR.
Ask the model to write code that checks your policy, then add that code behind a simple validation hook (e.g. "check your work by running 'just validate'") that the harness knows to always run after changes? It suddenly becomes the most law-abiding citizen ever.
Perhaps, but the majority of executives do not understand code at all. The scenario in the paper is realistic for a non-tech company suddenly adopting AI for daily business tasks.
HANDBOOK.md is a benchmark for long-context agentic instruction following, modeled on how enterprise employees follow company handbooks in their day-to-day work. Each task is a unique RL environment with internal tools and external MCP servers, spanning five enterprise domains: Finance, Medical Billing, Insurance, Logistics, and HR.
The prompts reflect the actual jobs enterprise workers perform every day. Each task drops an AI agent into a live company environment, requiring them to cross-reference an extensive, multi-section handbook against a cluttered inbox, a multi-channel Slack workspace, Jira queues, and a stack of files (spreadsheets, PDFs), and working out both what to do and what the handbook forbids.
It was copy/paste from my phone and when I posted there was no context / other comments and the github link was buried in the footer of the PDF of the paper.
This is a problem with long context models. To put it as simple and as bluntly as possible: just because they claim you can use 1M tokens in your context doesn't mean its true and you should do that.
Due to extreme quantization of models and the context's KV cache, and also just really shitty samplers provided to the user (hell, most are just getting rid of sampler knobs altogether), this problem will absolutely continue.
Want it to go away, almost like magic? Local inference. When its under your control, and no longer being forced to hold it wrong, all of the common LLM defects will go away.
> Want it to go away, almost like magic? Local inference. When its under your control, and no longer being forced to hold it wrong, all of the common LLM defects will go away.
This is just not true. Any local LLM you can host on consumer-accessible hardware has all of these defects, too. Adjusting the knobs doesn’t solve everything.
The closest you can get to frontier performance is Kimi K3, but you’re not hosting that unless your budget is on the order of a nice house in a good metro area.
I like my local LLMs as much as the next person and my office is currently uncomfortably warm from the amount of compute happening, but I would never agree that local LLMs solve all of the common LLM defects. This is peak wishful thinking.
In my experience, the local models and even the larger ones that we can’t run at home suffer more from long context degradation than the frontier models. You are exactly right that you need to manage context length, but even at fp16/bf16 the local models have a lower ceiling for usable context length in my experience.
This is a bit of a strawman. The parent comment wasn't suggesting that the inherent defects magically disappear nor was it suggesting that local inference today is sufficient for all tasks.
At the heart of it, self-hosting liberates your use cases from all the horribly opaque configuration, shadow prompting, etc. And local models are only getting better and more diverse every month.
>The parent comment wasn't suggesting that the inherent defects magically disappear
that is almost exactly what they said though..?
"Want it to go away, almost like magic? Local inference. [...] all of the common LLM defects will go away.
"[...]"
-----
edit: I suppose I have to spell this out.
"[...]" = "When its under your control, and [you're] no longer being forced to hold it wrong," ≈ "liberates your use cases from all the horribly opaque configuration, shadow prompting, etc."
and
"Almost like magic" is cliché hyperbole. It also means "not magic" in the same way that "almost like a dog" means "not a dog." It was immediately followed by the explicit clarification that was surgically omitted above, in absolute and conscious bad faith.
Hence, "[...]"
i omitted it because it doesnt change my point at all and it makes the quoted text smaller. not because of "absolute and conscious bad faith".
people can just read the original from the comment. its not like im quoting from a different website and trying to hide something. look 2 comments above mine and it is right there in full. anyone reading my comment would have seen the original comment already...
i have no idea why you took such offense to my comment, but you're reading way too much into it.
> "Almost like magic" is cliché hyperbole. It also means "not magic" in the same way that "almost like a dog" means "not a dog." It was immediately followed by the explicit clarification that was surgically omitted above, in absolute and conscious bad faith.
Incorrect. The omitted text doesn't change the relevant meaning of the quotation in the slightest. Why are you taking such offense?
> Want it to go away, almost like magic? Local inference. When its under your control, and no longer being forced to hold it wrong, all of the common LLM defects will go away.
> "Want it to go away, almost like magic? Local inference. [...] all of the common LLM defects will go away.
There's no difference between these two statements in the relevant context of the article, which is specifically "long policy documents do not reliably govern agents", or the comments above, which are specifically addressing how that issue is not solved by local agents.
I get your reading. My interpretation is that they’re claiming enshittification and that the local models don’t have that.
It’s an understandable view, but I’d be astonished by any local model processing very long context better than any frontier model (and now many racks are we talking).
> This is a bit of a strawman. The parent comment wasn't suggesting that the inherent defects magically disappear
The parent comment literally said that the common LLM defects would go away.
Direct quote:
> all of the common LLM defects will go away
Clearly it was meant that they will go away only if you use a custom harness, without the defects that now exist in all vendor-controlled harnesses, because of the non-aligned goals of the vendor and of the user.
While it was not spelled explicitly, it should be obvious that nothing can be improved if you run locally an LLM in the same way as the vendor. But you should not do that. It is easy to use better methods for sampling procedures and for prompt and context handling.
> Want it to go away, almost like magic?
True, almost like magic and magic are not the same thing, but I'd hesitate to call this a strawman.
[dead]
Do you work for a frontier lab by any chance?
No.
We're on the cusp of Kimi K3 becoming usable on sub-10k hardware.
https://github.com/gavamedia/deltafin
14 seconds per token? Not tokens per second. Seconds per token? That’s nowhere near the cusp!
At that rate, it would take me a mere 15 days to generate the number of tokens I typically use in a day.
"Yes, I'm using AI to speed up development. I'll submit that PR in two weeks time!"
That speed is significantly lower that what can be attained with streaming from SSDs.
Not only in big desktops, but even in most recent mini-PCs, it is possible to read concurrently from one PCIe 5.0 SSDs and one PCIe 4.0 SSD, at a total sustained reading throughput of around 20 GB/s.
With an optimized inference implementation, it should be possible to overlap completely the computations with streaming weights from SSDs.
This should improve the inference speed to around at least 1 token per second, on a cheap computer, under $2000 even at the current super-inflated prices.
There are enough tasks where this would be useful. Obviously one should use for most tasks a fast small LLM and use the big one only when this actually saves time.
It's need it to be an order of magnitude cheaper, but for some queries ("What to discuss at tomorrow's meeting") I can wait 12+ hours.
I'd rather just use my actual human brain to compute the answer at that point. I don't see the value at throughput that is this low.
I'd rather not be thinking about tomorrow's meeting all night! Just trust "the robot has got this"... and scramble in the morning when the robot has failed. But at least I got a good night's sleep!
For values of "usable" that include "14.6 seconds/token". It's a cool accomplishment! And newer hardware would speed it up some. But I think I'd want something a bit faster before declaring it usable in practice.
At my org we've been building AI agents and one internal rule we have is to use at most 50% of the models context window with the recommendation to not go over 25% for large context window models.
Anytime I see a "1M Context Window", my brain always goes "Gotcha so a 250k usable window"
Every single analysis I've seen done by anyone has basically come to the same conclusion: assuming sane BF16 or Q8 model quant, and BF16 or Q8 KV context quant (ie, not intentionally screwing over the model), you get about 250k before it pukes.
Why 25% and not 12% or 40%? Is this an arbitrary vibestimate or you had some tests done that pointed you to 25%? I am genuinely interested in how others deal with context issues. Also I would think the usable context window is variable depending on the task, for example summarizing documents vs analyzing large, scattered and complex instructions.
250k because thats what the smaller models supported and theres better training data in that part of the window
there have been some papers suggesting that the useful context is even smaller, and stays fixed as you change the context window size.
as a more general case though, i think the possibilities for what youd need to include in training to have many different paths of text be well represented enough over the long window means the later tokens will almost always be a lot more random than early ones?
there's a sheer amount of bits problem.
>in that part of the window
What matters is the relative distance. Using the latter 250k should be equivalent to using the first 250k.
I'm not the above poster, but my experiences generally line up with theirs.
The first 20% of context is usually very solid. Agent stays on task, is responsive and focused on the original goal.
The last 20% of context is usually pretty bad. Agent tends to loop, repeat, drift away into sub-goals (or non-goals).
The middle 60% is a bit of a grab-bag. Sometimes it's fine, sometimes it's not. It varies by task, prompt, and luck.
The above 25% and 50% limits seem like reasonable and simple rules to keep agents in the productive zones of usage.
It's generally not worth doing extensive testing to try to find your exact threshold unless you're running exactly the same prompt at scale, on similar data.
If you are in that spot, I won't actually recommend langchain tooling (I've been somewhat frustrated with it) but I think the concepts are worth understanding.
You can do something like https://docs.langchain.com/oss/python/langchain/test/evals. Trajectory evals in particular seem to suffer as context length grows (https://docs.langchain.com/oss/python/langchain/test/evals#t...).
I've used this for testing an agent that handles provisioning and permissions for developers with SaaS tooling (ex - if a developer asks, go provision an account [saml/oauth/etc] with the 3rd party, configure access, and let them know how to use it).
I'm about 6 months out of date at this point, so I won't speak for the latest models, but what was available early this year absolutely suffers from extended context lengths.
In my own experience (both Claude and Gemini 1M), you start running into problems around 60-70%. A 25% ceiling sounds like total cargo culting to me.
Benchmarks show a large drop in quality as context grows. Your opus will be acting like haiku above 300k tokens.
How do they go away with local models? It's a bug of all LLMs not just cloud vs local. As mentioned in another comment, they did test local models here too and those failed as well.
As parent implies, they're testing the wrong control mechanism. Why are you using policies instead of real controls over the weights and inference pipeline?
Well the answer is that VC-backed companies decided AI is not a domain expert tool for highly competent technical users, it's a magic oracle for the lowest common denominator. So you don't get any of the actually useful controls, just context engineering like that's fucking sane at all. It's like trying to program by navigating a git history. Not writing any new code, you don't have the ability to do that. No, you exclusively have the ability to move around a git history and cherry pick things. It's insulting that they want to charge money for this shit.
> just context engineering like that's fucking sane at all.
Meanwhile every model is at a battle to figure out how to stop you from jailbreaking it, which I assume takes a toll on your ability to prompt. Heck, if you read r/Claude and related subs during the early release of Fable there were a lot of complaints that it would downgrade you heavily.
> Well the answer is that VC-backed companies
Look, I enjoy local LLMs as much as anyone, but I think there is some motivated reasoning happening in this thread to try to make local LLMs sound like a utopia against those evil VCs.
Local LLMs suffer from the same problems.
And I think you didn't understand what I wrote, so let me reiterate: Of course policies are equally ineffective at strictly governing the behavior of local models. Why would that change? The actual problem is that somebody is attempting to misuse them for that in the first place. There are only two reasons for it:
- They have no other options
OR
- They have no idea what they're doing
Local models solve the first problem, and let you apply the appropriate tool by virtue of the control you have. The alternative does not.
In-context, your reply reads like you're asserting that anthropic's offerings give you all the same control as a local model, and that having this baseline of control over a computer program is "utopian". To me that sounds more like the second problem, and nothing can help you there.
> In-context, your reply reads like you're asserting that anthropic's offerings give you all the same control as a local model, and that having this baseline of control over a computer program is "utopian".
No, I’m talking about the idea up thread that having control over more settings removes all LLM defects with local models.
I have all the control I want over the local models they run. No combination of settings is going to make them outperform Anthropic’s models.
He didn't say that, and I didn't say that, because in fact neither of us were talking about just settings, so that's a pretty curious idea to be addressing.
> No combination of settings is going to make them outperform Anthropic’s models.
And even this is addressing something nobody was saying, and manages to come off as a marketing read.
> In-context, your reply reads like you're asserting that anthropic's offerings give you all the same control as a local model
It absolutely does not.
And what is the difference between cloud modeld vs local inference regarding this? It is still unclear in this thread - asking out of curiosity.
The ultimate difference isn't necessarily down to remote vs local. You can run a "local" model on the cloud the same way you could on a local machine and have all the same control over it. You can modify the weights, inject behavior, stack on arbitrary LoRAs, use whatever runtime you want, etc.
The problem arises in the case of anthropic, openai, etc. where you can't do those things. I think they have LoRA services that they'll charge you for, but that's about the extent of it. Think of it like having a compiler and its source code, and the operating system it runs on vs simply having access to invoking it via a web service.
Changing the weights and controls does not alleviate the context limits causing LLMs, local or cloud, to forget over time. I was asking how local LLMs would amelioratile this but looks like there is no solution.
One of the biggest fixes I've seen is just getting rid of traditional sampling. But first, let me say something about quantization, just to get this out of the way.
Like, lets say you already did the sane thing, your model[1] is already either FP16 or Q8 (and quantized by a competent practitioner of the art, ex: unsloth or bartowski), and your KV context is already FP16 or Q8... which means you now are already ahead of the major companies.
Google, OpenAI, and Anthropic heavily compress both K and V to insane levels, which might not appear to be so bad on short prompts, but especially with thinking enabled, its sort of the equivalent of JPEGing a JPEG repeatedly. Every time the model thinks, and records its thoughts into the context, and then reads it back later to think more, it becomes further and further imprecise.
All models with heavy KV context compression go off the rails somewhere between a quarter and a half of a million tokens. Every. Single. One. Every team that releases a model that has a limit of a quarter of a million did this on purpose, and it was the smart thing to do.
Now, lets say you dip your toes into samplers; this includes stuff like temp, top k, top p, min p, etc. I won't describe what they do, there are already good ELI5 articles out there to help you with that. They are, however, the original samplers, and the only ones the big companies use. None of them use the newer samplers that massively outperform them.
You know what you get with most providers? Temp as a knob, and it only goes between like 0.0 and 1.5. What if you want higher? Nope! What if you want to tune the other knobs? Usually no, too (OpenAI seems to still offer it on their higher end API plans, but Google has eliminated all knobs, and Anthropic apparently removing everything but temp in the future). What if you want other samplers? Not allowed.
Even restricting yourself to normal samplers, what if I wanted temp of 100, and min_p of 0.9 and no other samplers? That produces sane results for creative writing tasks, yet I could never do this, even I was paying for some $200/mo plan at Anthropic/OpenAI/Google/etc.
All of these samplers also cause a sort of JPEGing a JPEG repeatedly sort of error, it is the third source of it (model quant and KV cache quant are the other two). Long context insanity is probably caused more by sampling error more than it does by model and KV quant error.
What other samplers are there? Llama.cpp impls dynatemp, mirostat, top-n-sigma, and some others.
The one that I think more people need to look at is top-n-sigma. Temp and top-n-sigma alone has produced results that, even on ridiculously complex and purposefully tricky prompts, let models that have 1M native context happily go to the very limit of it without any signs of tell-tale degradation.
Want to go have fun with your new found freedom? Get Qwen 3.6 27B in Q4_K_M from a reputable dealer, set llama.cpp to do Q4 KV (yep, after I just said don't do that), and then run it with `--samplers "temperature;top_n_sigma" --top-n-sigma 1.0 --temp 1000`, and then compare it to the normal recommended values of `--temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.00`.
You will find that a lot of problems suddenly go away: long context degradation vanishes (which Qwen 3.6 will still do inside it's 250k context, especially when both model and KV are below Q8), forgetting what you said or it said earlier goes away, losing the plot half way through goes away, overreliance on cliches (Qwen has its own form of Claudisms, but are more subtle and less grating) also goes away, hallucinations happen far less, and lazyness also goes away (Qwen 3.6 has no real lazyness defects, but Gemma 4 does).
I have tested those top-n sigma settings on code generation as well. It is better than stock, and better than the commercial offerings by any of the American providers, but I still don't think LLMs can replace human programmers: it still can't think nor reason.
But yeah, moving to more modern samplers has done more to unfuck LLMs IMO than anything else you can do.
[1]: Assuming it isn't a native 4 bit model of some kind; quantizing them correctly to work in a local inference engine without actually quantizing anything is a bit of a PITA. See unsloth's work on the QAT Gemma 4 releases.
The needle benchmarks show, that models extended context works for the part, that can be explained as: "I can access/adress that part of the input".
I have no idea, why in that context, the number of attention heads isn't mentioned. Models have a limited set of them and obviously, a model can focus at N max things at a time, which has to put an upper bound of long context support in some way. There's just more things to lose focus to (or, mismanage the limited attention heads resources - per token)
There are no "attention heads" or fixed number of things a model can pay attention to ... or at least not exactly.
After every prompt the model decides "I have a weight of 1 to distribute between every token in my context". If you have ten tokens, each gets a weight of 0.1 ...
... except it's not that simple, because the LLMs don't distribute that "attention budget" equally. If your prompt was "where is Paris", then any tokens in context it can associate with Paris will get a greater share. If the word Paris is in your context, it might get 0.3 or 0.4 weight, and close by tokens might get 0.2, while other (unrelated) tokens get 0.03 or something.
Now, add lots of context, and you start to see the problem: more context = greater distribution of the attention budget. Even if the prompt is about Paris, and the Paris tokens get higher weights, they are only getting (say) 0.002 ... while unrelated tokens are getting 0.001.
All LLM "answers" are just math, computation, based on the context and those weights. If it can't "focus it's attention" because it's distributed among too much context, it's far more likely to miss the relevant tokens (eg. the Paris ones) and give you an answer that ignores key parts of context.
But again, there's no fixed number of things it can pay attention to: it's a gradual degradation of the chance of it seeing key info it should, based on the amount of context.
EDIT: Ok, I guess there are an internal thing called "attention heads"; TIL. They work exactly as I described (essentially I described a single "head").
Still, what I just wrote remains true: whether you have 1 head or 96 or ... however many your particular model has ... the fundamental issue isn't the number of heads. It's not like the LLM can pay attention to exactly (say) 96 things at once: each one of those heads has all of the available context "competing" for attention as described, and then the heads themselves "compete" to produce the answer (although not through the same mechanism; it's more like the weighting between heads is baked-into the LLM).
At the end of the day, the problem isn't "there aren't enough heads to pay attention" and the LLM hits a fixed limit. The problem is each head is trying to weight the relevant stuff, and the more stuff there is to distribute weights through, the less likely it will get it right (and then when all the heads are combined to produce the final answer, it misses key context).
> There are no "attention heads"
Yes there quite literally is internally in an LLM.
https://bactra.org/notebooks/nn-attention-and-transformers.h...
Just because something is called an "attention head" doesn't mean the terminology makes sense.
If I'm reading the article right, hungryhobbit is accurately describing a single attention head.
And... what having multiple attention heads means is that you do the "single attention head" thing several times, and average the results. There is no part of hungryhobbit's comment that changes if you make this adjustment.
In particular, he is clearly responding to this clause in the comment above:
>> the number of attention heads isn't mentioned. Models have a limited set of them and obviously, a model can focus at N max things at a time
Those are the "attention heads" that hungryhobbit is saying don't exist. You'll notice that this definition of "attention head" does not correspond to any part of an LLM; I fail to see the problem with saying that "attention heads", as defined by this clause, don't exist.
There quite literally are internals that are called attention heads in modern LLMs. Saying 'there are no attention heads' is objectively false, which definition you follow is irrelevant.
Imagine the following conversation:
How was your trip to Los Angeles?
There were no movie stars!
Is the second person's statement "objectively false"?
Or is it more likely that you, Catloafdev, enjoy intentionally misinterpreting simple statements?
You might notice that in the remainder of the sentence you quoted, hungryhobbit takes the trouble to define the attention heads that don't exist.
[dead]
Off topic, but I'm kinda sad there's only one Cosma Shalizi in the world. That dude is a treasure, I've been a fan for decades now.
Llm's have come on quite a bit beyond the fundamentals nowadays. Attention heads really are a thing.
One of my early experiments last year with open source models and context size was with GPT-OSS 20B (the mxfp4, the "smart" 4-bit quantization). Even though it boasted a 128k context size it was bad at recall around 32k characters (didn't bother to implement the tokenizer for counting).
The recall text was a simple hash generated, filler text from a dictionary file and a request at the end to return only the hash from the beginning of the prompt. Past 32k characters the response contained hallucinations of characters or full hashes.
Just having large context size doesn't paint a full picture of capabilities, prompt adherence and other quality metrics.
This is conspiratorial speculation though. They did test many open weights models which spanned the full spectrum of performance: Nemotron 3 Ultra second-worst, GLM 5.2 top five https://arxiv.org/html/2607.25398v1
Just need the hardware industry to catch up with consumer demand. 200kUSD+ on a computer capable of local inference is simply a non-starter.
>Want it to go away, almost like magic? Local inference.
Ah yes, magic that costs the same as a new car.
Not necessarily!
If you have a semi-recent MacBook with even 32GB, you can run 20GB models that are pretty damn smart, with room to spare for the rest of your toolchain.
If you’re reasonably connected to the code you’re writing and prompting the AI at the level of the code, not the level of the feature, you can get some fantastic results.
Sure, it’s not the completely automated dreamland that’s been sold, but it’s still a speed-up on par from going from assembler to a higher-level language, which is still immense.
And for effectively free, if you have a machine that would otherwise have been considered “developer-grade” for a lot of tasks anyway.
I tried this on my work M4 with 48 GB of memory and it's not as useful as I had hoped.
If you set a large context window you end up using all your memory. Prefill is slow. And the 10-20B models are somewhat poor at understanding intent and tend to think in circles, so if you set a small thinking budget to control context size the model ends up answering based on some half formed thoughts.
Or something like batch document summarization or image classification or whatever yeah it's great. But for "agentic" and coding workflows I think you would need a completely different harness setup and prompting style compared to what we are used to from the big proprietary models. I certainly have not yet figured out what that is.
What models have you found to work for which tasks? I find the local models very useful but not in a way that's replaced cloud models (yet, I remain hopeful).
A good setup will cost you the same as a decent house in a major metropolitan area. And then the price of a used car to upgrade your electrical to handle that kind of load, and get the cooling setup you need.
Tell me where I can buy a car for $1k or $2k.
It seems like you mean to use your own agent (where you control the prompts), not local inference. A custom agent works just as well with cloud based models.
I've noticed this trend of the sampler knobs being removed. Can you explain why this might be the case?
Would you mind sharing some of your setup? Which model and which params do you tweak (e.g. temperature)?
See the end of https://news.ycombinator.com/item?id=49100144
Long-context models do work as advertised... just not when combined with multi-turn conversation sessions.
It's my understanding that, for all current models having long-context capability, the "parts" of the model that allow them to do long-context processing, are mostly-untouched descendants of academic model architectures; where these academic models were designed around a very constrained use-case assumption: that anyone using more than a "normal" amount of context would specifically be trying to apply a reasonably-sized prompt to an unreasonably-sized block of plain-old embedded data, as a single-shot conversation. (Consider a task like e.g. "summarize this entire book." Context = [prompt] + [text of book].) This is the use-case all these architectures were refined and benchmarked around. And even today, this is still basically the only way these models' long-context processing capabilities work.
---
But I think it goes deeper than that. It's not just about how we've made long-context work so far. It's about what "attending to context" really means, and what that implies about what we'll ever expect a long-context model to be able to do.
I would describe long-context models as seeming to have two distinct types of "attention state" used when attending to the layer-1 input-vector / context window:
- They have some small amount (e.g. 8-32k) of regular "high-quality" attention state. This is likely inherited from the base model they are trained on top of.
- They also have a large amount of "low-quality" attention state. This is what the long-context training process has created.
The "high-quality attention state" isn't limited to referencing the first 8-32k of the context window, to be clear. The high-quality attention mechanism is just limited to "choosing" 8-32k of arbitrary data samples, each from its own arbitrary position in the context window, to form a bounded-sized vector upon which high-quality attention is then computed.
(You could think of this as the high-quality attention mechanism computing attention over the entire context, and then doing top-K sampling, i.e. zeroing out all but the K largest-valued elements of the resulting latent vector, where K=8-32k. That's not what it's actually doing—that'd require paying the very quadratic scaling cost that prevents us from just increasing the short-context window size—but it's a useful naive model for what it's doing.)
---
It's harder to directly explain what the low-quality attention mechanism does, so let me try an analogy.
(While no one can yet give a definitive accounting of the abstract computational processes the matrix-multiply-ops against a given LLM's weights encode, we can still speak of possibility-spaces, or analogies via information/computability theory on what sorts of abstract operations the LLM "must" be performing to achieve the results it achieves. This is that kind of thing.)
Think of an LLM's evolved context-window attendance mechanisms as being somewhat akin to the LLM being a regular CPU program, that has access to a 1. block of raw addressable memory (i.e. the context itself), and 2. a block of pointers into that memory (i.e. the learned first-layer attention over the context.)
Our LLM program's logic, through mutations during training, gradually evolves into a form where it could be seen as implicitly declaring and using a growing set of variables: some bound to static addresses in the block of raw memory (i.e. static context-window positions), and others bound to static addresses in the block of pointers, where the raw-memory-address-value of the pointer (and thus what data from the context-window it will read as when "dereferenced") is the result of a dynamic calculation over one or more other variables (i.e. an arbitrary weighting of the layer-1 input vector, where the resulting vector element is used as a selector, in layer 2, among other data that got sampled into "passthrough" vector-elements back in layer 1.)
Under this analogy, a model's "high-quality attention state" is the set of "named" pointer variables that it directly references in its logic.
As should be obvious, there can only be so many of these; you really fundamentally can't scale them up, without literally having "more logic" that defines and makes use of them (i.e. scaling up the size-in-weights of the model's layers.)
A model's "low-quality attention state", on the other hand, is the block of pointer memory itself, which can be as large as you like. Presuming the computational substrate permits the logic to compute on the addresses of memory within that pointer-memory block and then double-dereference them to get context data (and it does seem to!), your LLM's evolved logic can do "dynamic random-access reads" of arbitrary parts of the long context.
But, of course, the model won't be doing anything special with that data. It can attend to it, but not in a way that's unique to that data. It pulled in such data through generic logic that doesn't "know what it's looking at", after all.
---
IMHO, this analogy helps to make clear that while giving a model a raw "large-context capability" might help said model to attend to more data, it doesn't really help in making a model able to attend to a larger prompt.
Just by information theory, an instruction-following capability must imply that something analogous to logic keyed on those instructions gets embedded into the weights; and that logic, in order to activate and work, presumably requires its own state — i.e. requires that individual values from the context are getting attended into additional top-level named variables.
And while the short-context attendance mechanism does that, the long-context attendance mechanism inherently does not / cannot.
And this means that there is likely a practical upper limit (in terms of practical model-weight size + latent-vector size + etc) to "how much prompt" a model can follow during a single inference step. Regardless of how much data the model can "see" on a given inference step, its active "program logic" is only defined in terms of so many concurrent stateful computations.
---
Thus: trying to use a current long-context model's "low quality" attention to attend to a large prompt is just going to result in nonsense. None of these model families have ever been trained to follow a million sentences of rules simultaneously when answering a question. They're just going to attend with their inherited bounded "high quality" attention to your prompt as best as they're able, while forgetting literally everything about the prompt that doesn't fit in that bounded "high quality" window.
And anything that allows current long-context models to appear to interact well in multi-turn conversations with memory + KV caching + etc layers in operation — while also making use of its "low quality" attention to attend to large data — is some kind of hack, and will break like a hack.
this is cope (i mean, it happens a lot, but is not the cause of what is seem on the paper).
I noticed even with plenty of context, the model sometimes chose a path that "fools" the attention layers to bypass some rules. you can notice that the rules are not ignored, they are sidestepped. it's very predictable emergent behaviour after you see it happen. with more context, it actually gets worse to the point you will see some things obfuscated even.
Any model with a good score on this benchmark would have a good claim on superhuman abilities. Humans are pretty terrible at being thrown a long policy document and being expected to follow it
And while we shouldn't anthropomorphize these models too much, I wouldn't be surprised if many of the core reasons for failures are similar. Working memory is a limited resource; you can only focus on so many things at once; reasoning depth is limited; and many real-world policies are not actually meant to be implemented in the same way they are written and have insufficient specification of edge cases
With humans, we usually do the equivalent of RLHF, both via "training" with simulated cases, and via feedback while on the job. You would never hand a newbie a 124 page policy document and expect them to correctly apply it on the first task, or to do it reliably in the first month
The challenge with comparing these things to humans, is that humans learn. A newbie might not respect your organization’s set of policies on day one, but what about 3 months in? Or 3 years? Meanwhile there’s still no reasonable mechanism for automatically fine tuning LLMs or adjusting their harnesses to make them better at completing your organization’s objectives more successfully. They’re still overwhelmingly governed by the shared weights and harness policies found to be successful for the average case.
Models learn. It just costs $10B and 1 year to do what a human does every night.
LoRAs or even full fine-tunes would be much cheaper than that, and with some investment in the right infra could be updated regularly. And at least LoRAs can be swapped in and out cheaply, making them usable in large-scale inference providers. But there seems to be limited appetite in offering this. Both Anthropic and OpenAI no longer offer fine tuning for current models
does lora do a good job at teaching the model new things that werent in the training data?
without trillions of examples of following instructions at a million context length, im not convinced the behaviour is in the weights to begin with
No. Lora is for tuning behaviour and how the model applies what it learned in training. Teaching a model new facts is still expensive.
There was a recent-ish paper[0] from Sakana AI about baking facts from a document corpus into a LoRA adapter. Claimed near perfect recall on very large needle-in-haystack testing. Haven’t tried it myself though.
[0]: https://sakana.ai/doc-to-lora/
interesting... thanks.
Is this because of catastrophic forgetting, or is that more or less solved?
Is it, alternatively, because loading your LoRA plus someone else's can't be done in the same batch, so inference is made harder or becomes impossible to do at scale?
Or something else?
This is not about learning only.
It is about context window too so no matter how much money you put to train your model it can still have degraded performance in following instructions but it will be better to execute the instructions they can follow.
It is also about the harness and how that can help drive the model and pick and choose what to include or not in the context.
These LLMs do not understand the project. For them any next word is good as long as it was picked by the token predictor. It does not have any way to understand but only to do. probability distribution over their vocabulary and if that vocabulary is tainted and lost parts of the original context what is a good candidate there will not match the intention of the initial project.
That's not a model learning. It is model replacement.
At least these costs are currently preventing the planets surface from being covered in paperclip maximizers for the moment.
"You would never hand a newbie a 124 page policy document and expect them to correctly apply it on the first task, or to do it reliably in the first month"
So the behavior policy should reside in the weights, not in some ever expanding KV cache context, like sticking it in the wrong orifice if the goal is to be reproductive.
Instead of putting the policy document in some tightly crammed RAM, shove it in the weights that already exist, online / post training.
It does make one wonder if there is some way to compute the change to the weights from the computed context (so that we can free it up) without essentially continued "pre" training RMAD after each communication round.
> Any model with a good score on this benchmark would have a good claim on superhuman abilities. Humans are pretty terrible at being thrown a long policy document and being expected to follow it
Hmmm, if AI has to grow in the workplace, it has to follow processes to the letter, yet claude code forgets by the second turn my "don't commit" prompt.
Of course claude code is a generic and crap harness over terrific models that can't fit bureaucratic processes, and is increasingly worse at doing so anyway since Opus 4.6 peak.
Exactly: The best way to deal with this for humans is to use procedural scripts for different tasks, referring to the relevant bits of the (declarative) documentation.
I would imagine that doing something similar (using agent skills for insurance) would work much better for AI.
isn't it because there are too many contradictions and ambiguity? the reason it works for humans is because we don't apply everything at once either.
No, it's more than that. We can't remember everything. I hired, say, two years ago; as part of my onboarding process I had to read a bunch of policy and procedure documents, which were full of stuff that I didn't understand because I wasn't really in the context yet. So at the time, to me, those documents were full of arbitrary text that I didn't really understand. Some of it was rules that I had to follow, but at the time I didn't understand why, so it's just arbitrary rules.
How many arbitrary rules can you memorize? Do you even remember them two years later? If you do, then we can get to your statement.
And your statement is true. Humans do not run every action through a memorized list of rules, to see if any of them block the action. We don't. We're not going to, either, no matter how badly the policy manual writers want us to.
Social effects are a huge part of being human. Quite often we just do what the people around us are doing. In general it's not a bad rule, but can quickly lead go mass casualty events when the crowd has no idea what's going on, or we're messing with dangerous things.
That's a great comparison, human vs ai on a wall of text.
The problem is that it doesn't fit the sales pitch of LLMs and agents - humanlike or better, repeatably, 24/7, for a fraction of the price, you just need to make sure that you give it all the rules.
Unfortunately we can't really have a meaningful conversation until the money vampires have left so we will need to reschedule this until after the bubble.
Yeah checks out with my anecdotal experience with Claude. It is pretty great at following instructions - for about 10 minutes, after which it seems to ignore things I told it before.
I have quite explicit and strong instructions (e.g. don't write massive comments, use existing functionality, etc.) in CLAUDE.md files which seem to get bypassed surprisingly quickly when doing real tasks. Yet if I tell it these things in a prompt during the task, it performs way better.
Result is I'm trying to resist adding more and more things to CLAUDE.md files which in some scenarios it does well but in other scenarios totally ignores and messes up.
This is not what the article is talking about. Its talking about policy documents not it forgetting something 5 prompts ago. In fact you adding things to CLAUDE.md is more what its talking about.
Conceptually the same thing though. Claude.md is sourced at the beginning of the session, so will be pretty far back in context, just like user prompts from the beginning of the session.
[flagged]
Claude is a next-token predictor, appending to a long text document. Prompts aren't an independent kind of thing from policy documents. It's all text in the backscroll.
No because the CLAUDE.md is included in the first prompt which is given priority in the context. On prompt 20, it will remember message 1 better than message 12.
I believe the correct static instructions are about getting it at the right starting point for whatever class of projects you're working on; not as a continued referencable or "HOWTO" of what it's doing. They're all just "grooming" the LLM for future instructions.
The coding harness is what's getting it to continually align to your current instructions.
This is very obvious with local models.
Ok so what is the correct way to tell it "I don't care what is happening, you must uphold these rules at all times"? If it's not any configuration of .md files?
> you must uphold these rules at all times"?
You need to let go of the idea that this is something LLM's can do. They can't. At best, they can bias towards rules conformance with more or less likelihood, but coverage and conformance both go down super-linearly as you accumulate more rules, more context, and more output in a session. That's simply the nature of how these tools work and you need to engineer your workflows around it if you want to use them.
If you absolutely need some rules enforced, you need to adopt some framework for validating those rules that then rejects, reprocesses, or repeats any session that fails to satisfy them. In the best case scenario, this is some traditional deterministic validator (like a linter, compiler, analyzer, exhaustive test suite, etc in coding) but if you need to process in stochastic space because its something rich and ambiguous like natural language itself, then you want to dispatch a swarm very narrow, task-focused subagents that each validate against a very constrained subset. (And prepare yourself to have those to fail sometimes too. LLM's are noisy and cannot deliver strict rule enforcement on their own.)
You need to make the rule concrete somehow. I call it a "control". So for example, instead of instructing it "always run tests before committing", you (or you have it) make a git commit hook that always runs the tests first and that refuses the commit if they don't pass.
In this case, it is an advisory control only, because the LLM can also unhook that hook. And of course, it could also just disable the failing test(s) with some bullshit reason. But it is far better than assuming it will comply every time.
Then there is the "hard control", which is the inviolable that the LLM cannot bypass.
You need to move as much as is technically possible to either hard or (failing that) advisory controls.
I've seen pretty much every model including Claude bypass commit hooks constantly if there's even a tiny bit of friction, instead of pausing and asking me to help fix why it can't run the hooks (usually something that needs to be done outside the sandbox like `npm ci`).
That's why it's critical to have these checks run in a context that the model can't bypass, such as via github actions that block PR merges. Generally works pretty well for stuff at work (where we have all this stuff set up as part of continuous integration checks), but it means that for personal stuff I have to set up quite a bit more infrastructure to ensure that models don't just skip running tests (which they LOVE to do).
Or just make it so that the commit hook can't be edited or bypassed by Claude?
>In this case, it is an advisory control only, because the LLM can also unhook that hook
you can in principle make a hook in the harness itself that will run an auditor prompt that checks it adhered to the policy, correct the model and also make it known in advance.
This appears to be what the claude code-review is doing:
https://github.com/anthropics/claude-code/blob/7ef6eec9d9ba8...
this only checks whether the code is in compliance with CLAUDE.md though
So you are essentially saying "you can't, you can only safeguard from effects of LLM eventually ignoring it"
I think things like CI, linters, hooks, etc started as defense against humans eventually ignoring it (low blood sugar, under pressure, never cared in the first place, etc).
These seem like they would naturally extend to LLMs as well.
(I'm not pro-LLM on balance, but the solutions of today feel similar for humans and LLMs.)
It is almost exactly like the controls from the auditing world, except as applied to AI's instead of people. In fact, having worked at Deloitte in the past is what led me to make the connection immediately
Just like humans, yes.
[flagged]
in the plugins I'm using, it's basically _always_ adding information to the context. You're not going to do it manually; you can't go back in the context and add it because that'll break the cache. The way your programming harness works is by constantly reminding the LLM of the tools available.
A good programming harness is basically a stack. A good stack keeps building each layer. You _cannot_ pull things off the bottom of the stack because that's an expensive cache hit; but you can pull things off the top. So what your harness should be doing: <tools> <rules> <user content> onto each request _then_, when you get to the next request or result, pulling that out if there's some change.
So you can see it's the cache that's either exponentially growing or having to cache bust to keep it fresh.
Ask it to come back with a filled checklist and hand it over to a different agent with a fresh context window (three lines model). Or make it collapse the context and get back to the checklist.
Subagents whose only job is to review the actions of your other agents for rule compliance? It works reasonably well for me in complex workflows using Claude Code.
Can I ask how you set this up? Like is there some way to have that run automatically, similar to “auto mode” for approvals, or do you have to invoke it regularly?
Sure! I use an orchestrator main agent whose only job is to run subagents through the workflow process I've defined. Part of that workflow is to invoke a review subagent at particular points to check the spec, the implementation plan, and the code for rule conformance.
The review subagent has its own definition and gets invoked with a specific target, so the context is very focused on just rule enforcement and I don't have problems with it skipping rules.
The whole workflow is packaged up as a plugin, but you don't need that to get this approach to work. It should be sufficient to have the rules you want enforced written out somewhere, and to either kick off a focused review agent manually referencing them, or do something like I did and have it be a defined part of your workflow (depending on the automation level you want).
Hooks..
CI runs. Local Git hooks. Cursor also has hooks built into their agent. Other agent APIs probably have something similar.
[flagged]
As a hobbyist, I find it difficult to figure out how to make Claude stick with some repeating things I want it to do after every major action, like re-evaluate the completeness of tests, update the documentation, etc. And CLAUDE.md/AGENTS.md definitely did NOT help there, sadly.
Hooks can be pretty useful for that. A hook when it is finished ”run tests suite and check coverage” ”check if your changes require updating the docs”
[dead]
What I'm currently doing in a large refactor, is I created a super-run script; the super run script is devided into super-dev (Setup dev), super-test (run all tests), super-build (build artificats), super-e2e (test all artifacts), super-deploy (deploy finished).
Each super's sub functions should _fail hard_, and each script should be highly detailed; of course I'm not doing it myself, but in small increments of directed work, it can build up the necessary harness.
What I get is a CI that just starts with "run super-run.sh" and that gives it context, then each sub script provides context depending on if it succeeds or fails. If it fails, the agent is provided what it needs.
It's basically, you have to design the products of the AI to give itself the context. Another technique I'm testing out is a parallel set of files like <subject-module>.js, <subject-module>.test.js, <subject-module>.md which get pulled up if the Agent is looking for a file.
Don't use the default harness, write your own instead.
Any way to do that AND use subscription instead of per-token pricing from SOTA providers?
If the harness runs on your localhost, but the inference doesn't, it usually means it's calling some API. Whether you want to break TOS of your provider like that or simply buy tokens from our friends from UTC+8 timezone is an open question.
This is the way. Making your own agent to have a sticky memory context that is prepended to every execution is necessary to ensure each task is bounded by those precepts.
The trick I'm doing -- the model is given a tool that runs a prompt in the current thread to consolidate it's working memory and identity (it has a memory tool bound to the agent persona). When the prompt ends, the parts of memory that are marked as identity are merged together into a new system prompt, then the context restarts with only system prompt and this tool call surviving. Then it just keeps going.
[flagged]
Doesn’t that blow your token cache hit rate and balloon your costs (essentially everything is billed at the input token rate, not the cached rate)?
If it moved a block like
<system_prompt>
<tools_etc>
<project_prompts>
<user_call>
<llm_response>
=>
<system_prompt>
<tools_etc>
//Removed project_prompts
<user_call>
<llm_response>
<user_call>
<project_prompts> //reinserted
<llm_response>
Cache would break but if you did instead
<system_prompt>
<tools_etc>
<project_prompts>
<user_call>
<llm_response>
<user_call>
<project_prompts> //Duplicated with new user messsage
<llm_response>
It wouldn't bust cache, it would just make your input prompts slightly larger. Technically inefficient as you're duplicating the same rules over and over but I imagine for a smallish checklist/principles that it is tremendously more efficient than a cache break every message
so first, we treat context like a stack. Things are popped onto the stack; if we try to remove anything but the top of the stack, we'll break the cache.
So a good harness should be popping tool prompts, user hints after every return from the top so the context of tools and user hints never repeats and are allways prefixed to the current user prompt.
I've never inspected any harness, but opencode with the dynamic context compaction plugin appears to do this well.
By the time it does 10 turns of tool calls it forgets about the nagging again. It can and does ignore direct instruction from the user prompt too, sometimes repeatedly
I’ve had a lot of success using the root Claude.md for a handful of high level application wide rules and directions (I keep it pretty small), module specific claude.md in subfolders alongside the code with more specific rules and direction, and a custom rules backed /code-review skill that enforces it all and catches anything that was missed during implementation.
This is exactly what I have and it doesn't work well
Most people don't understand that 'agentic AI' is a completely synthetic, force fed capability by extensive Reinforcement Learning on synthetic domain specific 'agentic' datasets on post training.
If the LLM wasn't post-trained to adhere to specific handbook, it just won't work. If the LLM wasn't trained on an use case the lab decided was worth making a synthetic agentic dataset, it won't work as well as you want.
There's a reason the main agentic task LLMs excel at are coding tasks, it's the way of working of the creators, and they understand intimately the flow and can train for it.
I believe the true way will be able to easily fine tune models on your agentic use cases, but it would require a big company to compile a huge dataset on it's way of working and I don't think anyone wants to be the first.
In terms of long context, accurate attention retrieval from early tokens is just impossible, given the expansion of RoPE encoding for the positions, or in case of Kimi that don't use it anymore, as well as deepseek, early context is heavily compressed you lose accurate information.
If people spent more time studying about AI and how it works, they would realize that the default should be to one shot prompt your task with a big, cached system prmopt, with an user prompt that is just dynamic data, specified to the cheapest model that can do the job.
Unless you really can't do this given your problem, you should try to make a graph of well defined, step by step oneshot prompts, and THEN if your problem still can't be solved with that, then you start leveraging agents.
Despite this giving better results, and being more cost efficient, is evidently too much work then just letting the AI do all the work.
> In terms of long context, accurate attention retrieval from early tokens is just impossible, given the expansion of RoPE encoding for the positions, or in case of Kimi that don't use it anymore, as well as deepseek, early context is heavily compressed you lose accurate information.
"Every gambler knows the secret to survivin' is knowing what to throw away, and knowing what to keep." - Kenny Rogers
Humans have limited context, just like AI. The difference is that humans - at least some of the time - can figure out which pieces are more likely to be important, and therefore prioritize keeping those in the context.
What do you mean by a graph of one shot prompts?
Most people jump straight to agents when what they actually need is a graph. Example: a mining company receives free-text reports from field geologists. You could have:
Geologist report -> LLM call extracts minerals we are looking for (you inject a db query result on the user prompt), locations, assay mentions and risks into structured fields -> LLM call classifies evidence into positive indicators, negative indicators and unknowns -> LLM call estimates deposit potential and confidence -> database lookups inject regional ore demand, nearby deposits, infrastructure and historical yield data -> LLM call combines geological evidence with business context -> LLM call generates an investment recommendation and rationale.
That's what I mean by graph. Every step is a separate LLM call with a well-defined responsibility, consuming the output of the previous node. Each node can be tested, benchmarked, retrained, replaced, or monitored independently. Why would you use an agent here? You can cache every single system prompt on each call making your total token output much cheaper than having a full 'output' only token generation workflow which is what happens with agents.
There is nothing to discover. The workflow is already known. The company already knows how geologists evaluate prospects. The company already knows what data sources matter. The company already knows what the final output should look like. You don't want the model deciding which tools to call, which reasoning path to take, or which pieces of information are important every single run. You want the exact same process applied to every report so results are consistent, measurable, auditable and debuggable. My default is: One-shot prompt -> if not enough -> graph of LLM calls -> if not enough -> agent. A surprising amount of enterprise AI is really just: Unstructured input -> extraction -> classification -> enrichment from databases -> decision support. Not: Unstructured input -> autonomous agent spends 20 steps deciding what to do next.
Agents make sense when the workflow itself is unknown.
If the workflow is already understood, a graph is usually cheaper, more reliable, easier to evaluate, easier to debug, and less dependent on whatever synthetic "agentic" behaviors happened to get reinforced during post-training. I am sure people default to agents mostly because it's less engineering work than explicitly modeling the process.
Thanks, that helped. I get it. Yea, that’s the harness executing the workflow with the LLM being called at the right time. That ensures the process is consistent no matter what, in contrast to the agent calling out to tools and possibly doing different things every time. Basically, standard code being in control rather than the LLM being in control. Fully agree with this model. We should use deterministic code when we want the same process or algorithm every time and choose LLM callouts when we want “fuzzy” processing that is not as deterministic.
This is a great point, and one that has been blowing my mind for a while. The public think of AI as this black box thing approaching human intelligence. They don't know that when you look under the hood of a lot of ai products you see a string of prompts that anyone could put together themselves. The companies are just putting them together in a workflow.
The general public doesn't understand how these things work, at all, and the marketing makes it sound like it is magical.
That's why this paper is important. They gave it a scenario very similar to what would happen if an AI-frenzied executive suddenly mandated that AI be used to do basic company tasks - a scenario that is playing out all over the place right now.
I jump straight to graphs (unless it truly cannot be solved with a graph), but then the stakeholders get upset that I didn't use agents. Doesn't fit their marketing plan, I suppose. I don't mind dealing with that, but I can see why most jump to agents in order to avoid the human conflict.
Can I have graphs as the default and fall back to agentic behavior if the prompt / task can't be mapped to a known process graph?
Sure. You could easily have the model choose a workflow/graph for execution and then call a tool that implements the execution engine for that. The tool then makes calls for model inferencing at various points in the processing.
I would call that a pipeline, which may be a type of graph, but might have communicated your point more concretely to begin with
I called it a graph because really its a DAG (Directed Acyclic Graph) , in my example I kept it simple so there was only one input to every node, but in more advanced use cases you might have a synthesis node that takes in 2 or more inputs making it more akin to an execution graph in Tensorflow for example
DAGs are used in scientific workflows like snakemake. I wonder if such pipeline engines are use in LLM space.
Isn’t this just software? But with an LLM at various points instead of a deterministic call to some function?
can downvoters explain? this has been by experience with these tools too.
i thought it was well known that claude code got good at coding because anthropic bought tons of coding data from companies like mercor.
There was an article a few years ago called "Lost in the Middle: How Language Models Use Long Contexts" https://arxiv.org/abs/2307.03172
From my experience this holds true to this day. It was one of my core observations for similarity to the limitations of human working memory on "Engineering for Bounded Cognition"
Richard Hendricks solved this decisively with middle-out compression
I didn't get the reference, but it looks like im going to have to watch that series now :D
It holds up really well. I’m envious you get to watch it for the first tome, enjoy!
What inspired him to take this novel approach?
Long policy documents are also a problem for human agents. Without special training no one will retain 180 pages HR employee handbook, fire codes, OSHA safety rules, FCC regulations, the US legal code.
If the stakes are high, e.g, proceeding in ignorance could lead to prison time, people will favor inaction, even if the policy technically permits a corner case. If the stakes are low, people will completely override policy for the path of least resistance.
So what is tre answer then?
I feel like “discretion” parties missing. Do we need some kind of special discretion model?
This is real. I was very angry that AI kept breaking the rules I wrote, so I asked Claude to crawl its own history logs. And then I found: every time after it breaks a rule, the chance of breaking rules goes up.
I feel this is like few-shot in reverse. Few-shot is supposed to be good examples AI should follow. But when it breaks a rule, we correct it, it keeps breaking, and this whole thing actually raises the probability of more violations.
I wanted to know if there is any difference between writing rules in the prompt, writing them in CLAUDE.md, or not writing at all. So I did some short tests before. I asked Claude to open brand new sessions, test different topics with the rules I want to apply. The result turns out to be: in a fresh session, no matter the rule is in a prompt or in CLAUDE.md, models (Opus 4.8, 5 or Fable) all follow it fine, across models. Even Opus 4.8, the one always violates rules in our conversations, does it well.
I suspect it's the long context that breaks rules. But simulating a long conversation experiment is kind of hard, I still haven't found a good way to test it. So seeing this paper now, it completely answers the question I was stuck on these few days.
Besides, something caught my eyes in this paper: sometimes the model does run the check by the rules, and it really finds the violation, but its narrative still insists on its original wrong output.
My current approach is same as everyone here: use a separate hook or post-check to fix things. Because if you let the model fix it during generation, its narrative or main generation part sometimes just rejects the rule error it found.
>. But when it breaks a rule, we correct it, it keeps breaking, and this whole thing actually raises the probability of more violations.
In Pre-LLM days the 'nearest unblocked neighborhood' problem, where patching out one issue just immediately runs into another issue, or a different path back to the same issue. Since the models can learn new long time behaviors it's difficult to change the behavior without changing the context quite a bit.
For Claude, I used a UserPromptSubmit hook running inject_rules.py which reads RULES.md from the disk and prepends the whole thing to every prompt. That helps the rules to stop fading as context fills because it is reinforced every prompt.
Sure, it uses tokens slightly faster in the prompt, but I find it reduces overall token use, you can use it with pro, but of course, nothing works 100% of the time, but it's better. Emptying the memory helps too to avoid Claude making up stuff that messes with how I want it to act.
The general gist of inject_rules.py is :
RULES_PATH points at RULES.md
reads it with encoding='utf-8-sig' so the BOM is stripped
wraps it in a JSON object — hookSpecificOutput.hookEventName = "UserPromptSubmit", additionalContext = a preamble plus the full rules text
the preamble is the line you see above the rules: rules are in force for this turn, run rule 33's five tests before raising anything unasked
prints that JSON to stdout, which is how Claude Code takes it in
on OSError it returns 0 silently — if RULES.md is missing or unreadable, nothing is injected and the turn proceeds with no rules
why this vs having some response hook check the next outputs against the bunch of rules, and injects only when it goes off track?
Just the first thing I thought of to reinforce the rules. I don't use all my tokens anyway (now), so for me, there's no reason to introduce a point of failure, just reinforce every time.
This article to me also implies that there are some potential issues with large Spec based development flows, which I haven't been able to pin down lately.
Specifically, having agent implementation drift from the Spec.
I have run into the same problem, and started calling the phenomenon vision drift.
I noticed that the issue tracker I built was well suited to address this problem, as it handles board time-travel. So a simple command like ":replay 4h" gives you a clear overview of how the workflow has evolved over time, and you can checkout previous states at will.
Wrote more about it here:
https://dev.to/ljtn/vision-drift-addressing-the-next-problem...
Drift is huge between any large spec and agent implementations.
I've done a ton of testing and the model doesn't matter, fable or sol still miss a ton of detail and drift.
I'm building http://engine.build which closes the gap and makes sure the implementation matches the spec.
It's not the same as the satisfaction you get when solving complex problems with code yourself but writing clear specs and thinking through the problem is still very satisfying to me.
I noticed this behaviour a few months back, I think I was using Sonnet 4.6 at the time... I have strict rules about comments in the codebase, this all for personal projects, and the reason I restrict comments is to keep the token count low.
At some point between the model i was using and the previous version of it, Claude started inserting massive comments with references to tickets and other tasks. All this while having specific directives on the CLAUDE.md
Since then I resorted to developing my crapware as if I was the floor manager of a vehicle assembly line, and I have a few highly-specialized sub-agents running errands around the main session, but only ever taking care of a single concern. The main session builds with the knowledge contained in things like CLAUDE.md but the sub agents make sure things like the no/low-comments directives are either enforced, or factored into the final product.
What really helped me was to run any .md/prompts through an LLM to find contradictions, duplicates or ambiguities, repeatedly. That led to agents much better following instructions.
I absolutely believe it.
Codex has been pushing things to my main branch all week despite me repeatedly telling it not to and adding to my AGENTS.md very clear instructions for creating feature branches and putting up a PR. It keeps doing it in spite of all that.
I'm probably going to need to enable branch protection on my personal projects... What a pain.
The rules in main prompt and context is not working well in any times with LLM. The only way is to test results and let agent validate its own result against the rules. It will be interesting to see such benchmark. How the different models behave.
If I want to be sure I'm using my own deterministic rules checkers as a tool, that has to AI run and update the result accordingly.
Hi I just read your methodology and I have a quick question about how the handbook are parsed and feed into the context window? The article mentioned that each handbook contains roughly 8K to 79K tokens of extracted text, and did the harness system use grep or search tools to find relevant chunks and feeds to the context, or did it just feeds all the pdf output to the model? There might by distribution bias between real world tasks e.g. Agents grep keywords from docs and only use the relevant chunks. So all the models of the overall pass@1 is relatively low compared to real world scenarios, that might not be the same precision that user experience when they actually handle the daily task? How did the benchmark bridge the gap?
LLMs don't really read documents like parsers do, from what I've observed, they behave like they first skim the document and find the section that they think are relevant to their task, they don't really try to read everything and hold it inside their context.
The solution to that is pretty simple: get to the point, list all the requirements that they must do complete for that task, constraints on what they must not do, and optional recommendations for them to follow. Leave things that they can figure out on their own out of these documents.
Shorter is better. This can be applied to human communication too.
Not always. If an instruction is short and vague, then LLMs tend to fill in the blanks by guessing and the results are unpredictable. Semantic density is the key, prompts/instructions should be actionable and unambiguous over being terse and vague.
So, it's more about being concise.
“It can scarcely be denied that the supreme goal of all theory is to make the irreducible basic elements as simple and as few as possible without having to surrender the adequate representation of a single datum of experience.”
For my own (rather idiosyncratic) harness I've been experimenting [0] with "compiling" long markdown specifications into small executable logic programs. It's too early to tell for sure, but I believe that this approach does have its merits when you want some guarantees about how your agents behave for longer tasks.
[0] https://github.com/deepclause/deepclause-sdk
Interesting idea. I’ve been noodling about something similar myself for a few months, but I haven’t moved forward with testing it. What sort of outcomes are you seeing with it? IMO, we’re never going to get to AGI without fusing “soft” AI decision making with “hard” logic and symbolic algorithmic reasoning. Humans don’t realize this most of the time, but we routinely use them all.
Sorry, did not notice your comment until just now.
So far I am observing two things:
1. For smaller models, performance on Benchmarks such as DeepPlanning does increase significantly.
2. Context hygiene for sub agents becomes much simpler, since that can be expressed relatively concise and the mechanics are handled by the runtime automatically.
Still looking for a good test cases to study possible advantages, but running reliable benchmarks does take time and money...
Are you the Deepclause author? I tried it yesterday and my first reaction was that it was slow. Perhaps I’m doing something wrong, however. Running against the same model in Pi was lightspeed in comparison. Second reaction is the prompt editor needs to handle more than a single line of text and it needs Emacs editing key bindings like Pi. I’m happy to do some testing on it and provide further feedback. What’s the best way to submit that? GitHub issues?
Yes, I am the author and thanks so much for trying! Please do submit a github issue. My first suspicion about the speed is that maybe an inner loop is taking too many turns until the model finally realizes that a task is finished (so that in turn the runtime knows whether the predicate failed or not and can continue execution accordingly). Happy to take a closer look!
The point about multiline prompts is very valid obviously, that's on the todo list.
Cool, will do. Was playing around with it last night some more.
This is what spec driven development tries to solve.
Multiple rounds of generating small contacts documents that grow from the original idea , trying to keep each slice small enough to process for a human to approve/disprove .
Eventually it leads to a long list of tasks grouped by functionality. You start a new context and the orchestrator agent dispatches tasks to sub agents with a limited amount of information provided to each sub agent.
Also should have adversarial review and approval gates with other agents and roles.
I have not seen any real life examples of successful application of SDD. I heard a lot of marketing BS, but nothing real. My own experiments with SDD produced very disappointing results even on very small projects.
I've had good results from SDD, and so have some of my teammates. The guidance I've heard over and over again is: the more you define, the better it does. I find that breaking things out into individual steps is helpful. However, your mileage may vary. Things like Gastown and GetShitDone, I have not found to be particularly useful.
If the authors are reading: Why did you let an AI model author parts like "Design Principles"? It's not good writing and it is obvious.
> Failures follow consistent patterns: agents let a plausible in-environment request override the standing policy
I prefer this behavior in a lot of situations. A plausible request can also be a genuine one.
Alignment at the grain of each tool response is way more effective than a static system prompt. I try to keep mine under 500 tokens. Why bother fighting recency bias? It's pretty much what you want most of the time.
Opus 4.8 (max thinking) scored highest and Grok 4.3 lowest
It's hard to understand what's going on with Grok. It's like it has capabilities in a theoretical sense but maybe the training is so focused on being in x.com/grok.com with the web search tool enabled for "is this true?11" type queries that with any API type usage with document workflow instructions, tool use, code gen etc it completely falls over
After they acquired Cursor, Grok 4.5 seems like a completely new model, performing at Opus 4.6 level, I'd say. But much cheaper and faster.
maybe it is Grokimi?
https://venturebeat.com/technology/cursors-composer-2-was-se...
Attention vs. Consistency
When "performance" breaks down over long lengths, one could attribute it to a lapse in attention, but one could equally suspect inconsistent instructions.
The fewer instructions and conditions that need to be simultaneously met the easier it is to comply, but with more and more instructions one is bound to introduce internal inconsistencies within the instructions.
Policy documents do not govern agents at all. Conformance is distributed and completely unreliable.
How many times have you told an agent not to do something then had to correct it?
You must always flip the frame. Objective analysis is way better with llms than steering via skills.
This is just a small example of why "loops" became popular for a minute and now it is "graphs"
Why would anyone think that models optimized for efficient context management, giving much more weight to a short sliding window, would attend to distant, heavily diluted tokens?
Plus the model's capacity to take more context into account and actually integrate it to the output is simply limited by the number of activated parameters. If you give it a playbook, you are forcing to choose it between attending to the playbook and the task at hand.
If you want to force it to work step-by-step, you need to present the steps one-by-one. Ideally with rules for the current step at hand and maybe relevant input again, depending on overall task size.
Why did you think models love to re-read files before editing them? It increases recall quality and thus edit precision and thus benchmarks.
> limited by the number of activated parameters
not sure I got it?
Separately, the frontier labs are kinda pushing us into that behaviour by releasing models with ever-larger context windows.
To run at decent speed, all models try hard to use only most likely relevant part of the context and most likely relevant weights (MoE) to predict the next token. Doing the math in full is unfeasible.
Control vectors for the win? It feels like the way to fix this is to pull out a control vector immediately after processing the handbook and use that to steer later inference. That should stop the drift over long distances, but it doesn't guard against the handbook itself being too big.
Evoking Gödel via GPT, with the requisite em-dash: "No sufficiently expressive formal governor can be simultaneously consistent, complete, and able to prove every relevant proposition—including all propositions about itself."
But "Read AGENTS.md, including the middle" sure helps.
Dealing with agents/LLMs based on "instructions & guidelines" has taught me - nothing (un)reliably governs agents other than agents themselves or (rather i.e.) their motherships (assuming they can and they intend to). Or if you add ton of local tooling.
Having only dipped my toes in generative ai recently I'm surprised how small even a 1mio window is. A semi serious project can take several session in one sitting. Especially as degradation sets in waay before the window is full.
Few pointers:
Dont try and handle the entire project in context.
Use a well structured filesystem layout for your code with a few lines in an AGENTS.md describing the layout and core architectural requirements (no more than that, as per the article!).
Then work on small-medium tasks at a time with a fresh context.
At the end of your task, ask the agent if there are any key points about the project layout or architecture it would want to add to memory - audit those manually and amend your AGENTS.md accordingly.
If your code is well structured and you keep your tasks localised, you can get away with seemingly minuscule context windows.
It's also worth noting that high effort models love to slurp up whatever context they can. You almost never need/want high effort for non cross-cutting tasks.
Good points, but it still seems to me that the technology is far from mature.
Of course - its only a few years old :) But I think the biggest problem is that no-one directly tells you how to use the technology effectively. Mostly because no-one really knows, hence the reason for all the competing harnesses, agents, etc - the models are all just intelligence on top of that tooling.
I don't understand why a model has to follow the instructions. Don't get surprised when it shows its true color. Plus, do users (not the researchers) really check whether the response follows the instructions?
Models don't have to follow instructions, but during RLHF that is one of the things they are scored on so a premise of the idea is part of the model, but it's also balanced on accomplishing the end goal. They model may determine, correctly or incorrectly, that your rules suck and do what it thinks is best.
I got pumped seeing the OKF format from Google (which is just a standardization of wiki pattern) but quickly realized it could not yet combine many subtle concepts together in an efficient way.
Are adversary agents a solution here? I've had good success using them to prevent the base agent from going off plan.
Yeah I feel like the longer it is, the more likely they are to just ignore it. With LLMs shorter is better. And nothing is guaranteed lol.
LLMs are not people. Employee handbooks, like all other social rules, are not self-executing; they depend on the fact that human beings are innately social animals.
Tell that to the executives who are looking to replace people with AI.
> Handbook.md shows that long policy documents do not reliably govern agents
Serious question: why would anyone expect otherwise?
This is a problem with soft rules and LLM in general, and it makes sense that it gets worse on complicated tasks with long context.
Glad to be able to put some numbers on it.
In my experience, the more structure you enforce on models, the worse they perform and the less they actually do what you want.
yep all the advice about context engineering, harness whatever is so silly. ai doesnt give a flying fuck about some IMPORTANT instruction in your claude.md.
It does what its has been trained to do. So find out what its trained to do and just use it to do that. This is not general intelligence.
Yes. And all the “context engineering”, document/decision frameworks etc just make it way harder to work with the ai. You put in a lot of work just for an illusion of control.
simonw has been correctly asserting that there is no deterministic way to prevent hallucinations. I’d argue that this obstinance could be the seed of a strong argument in favor of their capacity to eventually coalesce some form of consciousness
I guess that means my code is conscious cuz it often doesn’t do what I want either :)
I'd be very interested to see a comparison with actual humans on the same benchmark.
This is why the future is in deterministic static analysis and policy-as-code frameworks.
Policy-in-English? Model implicitly complains that it's TL-DR.
Ask the model to write code that checks your policy, then add that code behind a simple validation hook (e.g. "check your work by running 'just validate'") that the harness knows to always run after changes? It suddenly becomes the most law-abiding citizen ever.
Perhaps, but the majority of executives do not understand code at all. The scenario in the paper is realistic for a non-tech company suddenly adopting AI for daily business tasks.
One question I'm asking myself is: can LLMs be bored?
Confirmed. These are autistic geniuses with ADHD. Handle with that care.
Sounds about right
Hard to take this seriously when it has all the hallmarks and annoying tics indicating it has been written by Claude.
Well no shit obviously. Just cause you tell the random text generator to follow some rules doesn't mean it will.
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[dead]
[flagged]
[flagged]
[flagged]
[dead]
HANDBOOK.md is a benchmark for long-context agentic instruction following, modeled on how enterprise employees follow company handbooks in their day-to-day work. Each task is a unique RL environment with internal tools and external MCP servers, spanning five enterprise domains: Finance, Medical Billing, Insurance, Logistics, and HR.
The prompts reflect the actual jobs enterprise workers perform every day. Each task drops an AI agent into a live company environment, requiring them to cross-reference an extensive, multi-section handbook against a cluttered inbox, a multi-channel Slack workspace, Jira queues, and a stack of files (spreadsheets, PDFs), and working out both what to do and what the handbook forbids.
https://github.com/surge-ai/handbook/tree/main
Please don't paste walls of text into the comment field without quotation marks. It wastes all of our time.
It was copy/paste from my phone and when I posted there was no context / other comments and the github link was buried in the footer of the PDF of the paper.