Hacker News

> Want it to go away, almost like magic? Local inference. When its under your control, and no longer being forced to hold it wrong, all of the common LLM defects will go away.

This is just not true. Any local LLM you can host on consumer-accessible hardware has all of these defects, too. Adjusting the knobs doesn’t solve everything.

The closest you can get to frontier performance is Kimi K3, but you’re not hosting that unless your budget is on the order of a nice house in a good metro area.

I like my local LLMs as much as the next person and my office is currently uncomfortably warm from the amount of compute happening, but I would never agree that local LLMs solve all of the common LLM defects. This is peak wishful thinking.

In my experience, the local models and even the larger ones that we can’t run at home suffer more from long context degradation than the frontier models. You are exactly right that you need to manage context length, but even at fp16/bf16 the local models have a lower ceiling for usable context length in my experience.

How do they go away with local models? It's a bug of all LLMs not just cloud vs local. As mentioned in another comment, they did test local models here too and those failed as well.

At my org we've been building AI agents and one internal rule we have is to use at most 50% of the models context window with the recommendation to not go over 25% for large context window models.

Anytime I see a "1M Context Window", my brain always goes "Gotcha so a 250k usable window"

The needle benchmarks show, that models extended context works for the part, that can be explained as: "I can access/adress that part of the input".

I have no idea, why in that context, the number of attention heads isn't mentioned. Models have a limited set of them and obviously, a model can focus at N max things at a time, which has to put an upper bound of long context support in some way. There's just more things to lose focus to (or, mismanage the limited attention heads resources - per token)

One of my early experiments last year with open source models and context size was with GPT-OSS 20B (the mxfp4, the "smart" 4-bit quantization). Even though it boasted a 128k context size it was bad at recall around 32k characters (didn't bother to implement the tokenizer for counting).

The recall text was a simple hash generated, filler text from a dictionary file and a request at the end to return only the hash from the beginning of the prompt. Past 32k characters the response contained hallucinations of characters or full hashes.

Just having large context size doesn't paint a full picture of capabilities, prompt adherence and other quality metrics.

This is conspiratorial speculation though. They did test many open weights models which spanned the full spectrum of performance: Nemotron 3 Ultra second-worst, GLM 5.2 top five https://arxiv.org/html/2607.25398v1

>Want it to go away, almost like magic? Local inference.

Ah yes, magic that costs the same as a new car.

I've noticed this trend of the sampler knobs being removed. Can you explain why this might be the case?

Would you mind sharing some of your setup? Which model and which params do you tweak (e.g. temperature)?

Long-context models do work as advertised... just not when combined with multi-turn conversation sessions.

It's my understanding that, for all current models having long-context capability, the "parts" of the model that allow them to do long-context processing, are mostly-untouched descendants of academic model architectures; where these academic models were designed around a very constrained use-case assumption: that anyone using more than a "normal" amount of context would specifically be trying to apply a reasonably-sized prompt to an unreasonably-sized block of plain-old embedded data, as a single-shot conversation. (Consider a task like e.g. "summarize this entire book." Context = [prompt] + [text of book].) This is the use-case all these architectures were refined and benchmarked around. And even today, this is still basically the only way these models' long-context processing capabilities work.

---

But I think it goes deeper than that. It's not just about how we've made long-context work so far. It's about what "attending to context" really means, and what that implies about what we'll ever expect a long-context model to be able to do.

I would describe long-context models as seeming to have two distinct types of "attention state" used when attending to the layer-1 input-vector / context window:

- They have some small amount (e.g. 8-32k) of regular "high-quality" attention state. This is likely inherited from the base model they are trained on top of.

- They also have a large amount of "low-quality" attention state. This is what the long-context training process has created.

The "high-quality attention state" isn't limited to referencing the first 8-32k of the context window, to be clear. The high-quality attention mechanism is just limited to "choosing" 8-32k of arbitrary data samples, each from its own arbitrary position in the context window, to form a bounded-sized vector upon which high-quality attention is then computed.

(You could think of this as the high-quality attention mechanism computing attention over the entire context, and then doing top-K sampling, i.e. zeroing out all but the K largest-valued elements of the resulting latent vector, where K=8-32k. That's not what it's actually doing—that'd require paying the very quadratic scaling cost that prevents us from just increasing the short-context window size—but it's a useful naive model for what it's doing.)

---

It's harder to directly explain what the low-quality attention mechanism does, so let me try an analogy.

(While no one can yet give a definitive accounting of the abstract computational processes the matrix-multiply-ops against a given LLM's weights encode, we can still speak of possibility-spaces, or analogies via information/computability theory on what sorts of abstract operations the LLM "must" be performing to achieve the results it achieves. This is that kind of thing.)

Think of an LLM's evolved context-window attendance mechanisms as being somewhat akin to the LLM being a regular CPU program, that has access to a 1. block of raw addressable memory (i.e. the context itself), and 2. a block of pointers into that memory (i.e. the learned first-layer attention over the context.)

Our LLM program's logic, through mutations during training, gradually evolves into a form where it could be seen as implicitly declaring and using a growing set of variables: some bound to static addresses in the block of raw memory (i.e. static context-window positions), and others bound to static addresses in the block of pointers, where the raw-memory-address-value of the pointer (and thus what data from the context-window it will read as when "dereferenced") is the result of a dynamic calculation over one or more other variables (i.e. an arbitrary weighting of the layer-1 input vector, where the resulting vector element is used as a selector, in layer 2, among other data that got sampled into "passthrough" vector-elements back in layer 1.)

Under this analogy, a model's "high-quality attention state" is the set of "named" pointer variables that it directly references in its logic.

As should be obvious, there can only be so many of these; you really fundamentally can't scale them up, without literally having "more logic" that defines and makes use of them (i.e. scaling up the size-in-weights of the model's layers.)

A model's "low-quality attention state", on the other hand, is the block of pointer memory itself, which can be as large as you like. Presuming the computational substrate permits the logic to compute on the addresses of memory within that pointer-memory block and then double-dereference them to get context data (and it does seem to!), your LLM's evolved logic can do "dynamic random-access reads" of arbitrary parts of the long context.

But, of course, the model won't be doing anything special with that data. It can attend to it, but not in a way that's unique to that data. It pulled in such data through generic logic that doesn't "know what it's looking at", after all.

---

IMHO, this analogy helps to make clear that while giving a model a raw "large-context capability" might help said model to attend to more data, it doesn't really help in making a model able to attend to a larger prompt.

Just by information theory, an instruction-following capability must imply that something analogous to logic keyed on those instructions gets embedded into the weights; and that logic, in order to activate and work, presumably requires its own state — i.e. requires that individual values from the context are getting attended into additional top-level named variables.

And while the short-context attendance mechanism does that, the long-context attendance mechanism inherently does not / cannot.

And this means that there is likely a practical upper limit (in terms of practical model-weight size + latent-vector size + etc) to "how much prompt" a model can follow during a single inference step. Regardless of how much data the model can "see" on a given inference step, its active "program logic" is only defined in terms of so many concurrent stateful computations.

---

Thus: trying to use a current long-context model's "low quality" attention to attend to a large prompt is just going to result in nonsense. None of these model families have ever been trained to follow a million sentences of rules simultaneously when answering a question. They're just going to attend with their inherited bounded "high quality" attention to your prompt as best as they're able, while forgetting literally everything about the prompt that doesn't fit in that bounded "high quality" window.

And anything that allows current long-context models to appear to interact well in multi-turn conversations with memory + KV caching + etc layers in operation — while also making use of its "low quality" attention to attend to large data — is some kind of hack, and will break like a hack.

this is cope (i mean, it happens a lot, but is not the cause of what is seem on the paper).

I noticed even with plenty of context, the model sometimes chose a path that "fools" the attention layers to bypass some rules. you can notice that the rules are not ignored, they are sidestepped. it's very predictable emergent behaviour after you see it happen. with more context, it actually gets worse to the point you will see some things obfuscated even.

The challenge with comparing these things to humans, is that humans learn. A newbie might not respect your organization’s set of policies on day one, but what about 3 months in? Or 3 years? Meanwhile there’s still no reasonable mechanism for automatically fine tuning LLMs or adjusting their harnesses to make them better at completing your organization’s objectives more successfully. They’re still overwhelmingly governed by the shared weights and harness policies found to be successful for the average case.

"You would never hand a newbie a 124 page policy document and expect them to correctly apply it on the first task, or to do it reliably in the first month"

So the behavior policy should reside in the weights, not in some ever expanding KV cache context, like sticking it in the wrong orifice if the goal is to be reproductive.

Instead of putting the policy document in some tightly crammed RAM, shove it in the weights that already exist, online / post training.

It does make one wonder if there is some way to compute the change to the weights from the computed context (so that we can free it up) without essentially continued "pre" training RMAD after each communication round.

Exactly: The best way to deal with this for humans is to use procedural scripts for different tasks, referring to the relevant bits of the (declarative) documentation.

I would imagine that doing something similar (using agent skills for insurance) would work much better for AI.

> Any model with a good score on this benchmark would have a good claim on superhuman abilities. Humans are pretty terrible at being thrown a long policy document and being expected to follow it

Hmmm, if AI has to grow in the workplace, it has to follow processes to the letter, yet claude code forgets by the second turn my "don't commit" prompt.

Of course claude code is a generic and crap harness over terrific models that can't fit bureaucratic processes, and is increasingly worse at doing so anyway since Opus 4.6 peak.

That's a great comparison, human vs ai on a wall of text.

The problem is that it doesn't fit the sales pitch of LLMs and agents - humanlike or better, repeatably, 24/7, for a fraction of the price, you just need to make sure that you give it all the rules.

Unfortunately we can't really have a meaningful conversation until the money vampires have left so we will need to reschedule this until after the bubble.

isn't it because there are too many contradictions and ambiguity? the reason it works for humans is because we don't apply everything at once either.

This is not what the article is talking about. Its talking about policy documents not it forgetting something 5 prompts ago. In fact you adding things to CLAUDE.md is more what its talking about.

I’ve had a lot of success using the root Claude.md for a handful of high level application wide rules and directions (I keep it pretty small), module specific claude.md in subfolders alongside the code with more specific rules and direction, and a custom rules backed /code-review skill that enforces it all and catches anything that was missed during implementation.

I believe the correct static instructions are about getting it at the right starting point for whatever class of projects you're working on; not as a continued referencable or "HOWTO" of what it's doing. They're all just "grooming" the LLM for future instructions.

The coding harness is what's getting it to continually align to your current instructions.

This is very obvious with local models.

What do you mean by a graph of one shot prompts?

> In terms of long context, accurate attention retrieval from early tokens is just impossible, given the expansion of RoPE encoding for the positions, or in case of Kimi that don't use it anymore, as well as deepseek, early context is heavily compressed you lose accurate information.

"Every gambler knows the secret to survivin' is knowing what to throw away, and knowing what to keep." - Kenny Rogers

Humans have limited context, just like AI. The difference is that humans - at least some of the time - can figure out which pieces are more likely to be important, and therefore prioritize keeping those in the context.

can downvoters explain? this has been by experience with these tools too.

i thought it was well known that claude code got good at coding because anthropic bought tons of coding data from companies like mercor.

So what is tre answer then?

I feel like “discretion” parties missing. Do we need some kind of special discretion model?

Richard Hendricks solved this decisively with middle-out compression

Drift is huge between any large spec and agent implementations.

I've done a ton of testing and the model doesn't matter, fable or sol still miss a ton of detail and drift.

I'm building http://engine.build which closes the gap and makes sure the implementation matches the spec.

It's not the same as the satisfaction you get when solving complex problems with code yourself but writing clear specs and thinking through the problem is still very satisfying to me.

>. But when it breaks a rule, we correct it, it keeps breaking, and this whole thing actually raises the probability of more violations.

In Pre-LLM days the 'nearest unblocked neighborhood' problem, where patching out one issue just immediately runs into another issue, or a different path back to the same issue. Since the models can learn new long time behaviors it's difficult to change the behavior without changing the context quite a bit.

Shorter is better. This can be applied to human communication too.

why this vs having some response hook check the next outputs against the bunch of rules, and injects only when it goes off track?

Interesting idea. I’ve been noodling about something similar myself for a few months, but I haven’t moved forward with testing it. What sort of outcomes are you seeing with it? IMO, we’re never going to get to AGI without fusing “soft” AI decision making with “hard” logic and symbolic algorithmic reasoning. Humans don’t realize this most of the time, but we routinely use them all.

Models don't have to follow instructions, but during RLHF that is one of the things they are scored on so a premise of the idea is part of the model, but it's also balanced on accomplishing the end goal. They model may determine, correctly or incorrectly, that your rules suck and do what it thinks is best.

After they acquired Cursor, Grok 4.5 seems like a completely new model, performing at Opus 4.6 level, I'd say. But much cheaper and faster.

> limited by the number of activated parameters

not sure I got it?

Separately, the frontier labs are kinda pushing us into that behaviour by releasing models with ever-larger context windows.

I guess that means my code is conscious cuz it often doesn’t do what I want either :)

yep all the advice about context engineering, harness whatever is so silly. ai doesnt give a flying fuck about some IMPORTANT instruction in your claude.md.

It does what its has been trained to do. So find out what its trained to do and just use it to do that. This is not general intelligence.

Tell that to the executives who are looking to replace people with AI.

Perhaps, but the majority of executives do not understand code at all. The scenario in the paper is realistic for a non-tech company suddenly adopting AI for daily business tasks.

I have not seen any real life examples of successful application of SDD. I heard a lot of marketing BS, but nothing real. My own experiments with SDD produced very disappointing results even on very small projects.

Few pointers:

Dont try and handle the entire project in context.

Use a well structured filesystem layout for your code with a few lines in an AGENTS.md describing the layout and core architectural requirements (no more than that, as per the article!).

Then work on small-medium tasks at a time with a fresh context.

At the end of your task, ask the agent if there are any key points about the project layout or architecture it would want to add to memory - audit those manually and amend your AGENTS.md accordingly.

If your code is well structured and you keep your tasks localised, you can get away with seemingly minuscule context windows.

It's also worth noting that high effort models love to slurp up whatever context they can. You almost never need/want high effort for non cross-cutting tasks.

Please don't paste walls of text into the comment field without quotation marks. It wastes all of our time.