Hacker News

For me that moment was Gemma 4 12B QAT. You're not suddenly going to start throwing your hardest programming problems at Gemma 4 12B QAT, it is still 15B parameters less. It's more that, aside from pelican art which isn't what local models are for, I didn't see anything on Simon's post that it couldn't assist with or largely succeed at.

It can run 80-100t/s on a laptop, can understand images natively and do bounding boxes, read tiny text, understands audio natively as well and can transcribe or translate anything you say, can do accurate long context retrieval with pretty large context windows, tool calling, excellent reasoning and is very token efficient.

It's only 7GB including the mmproj or 8GB with MTP. The Qwen 3.8 27B model Simon was using is ~18GB with MTP+mmproj, rather than 17GB alone. The point is not really that you compare these models directly, but that Gemma 4 12B QAT was really a special moment in model releases deserving of a similar reaction relative to its size, but was mutilated by Google themselves, Unsloth and Llama.cpp.

The overall appreciation I think we're seeing this year in particular is that people are easily surprised when multiple things are improving simultaneously which produce seemingly exponential changes. It isn't just that models are getting smaller, or that reasoning is getting better, or that speculative decoding is becoming mainstream, or that models can understand audio and images better now, or that they can reliably call tools which expands their capabilities, or that context windows are getting larger, or that accurate retrieval is improved, or that.... and so on. It's all of them narrowing in at once that is starting to make local models incredible and truly useful for far more use cases on the existing hardware people already have.

Like the old proverb: "The marvel is not that the bear dances well, but that the bear dances at all."

Full agree. I until very recently thought AI tools of today were limited to prohibitively expensive high end hardware hosted in data centers.

I was surprised and amazed to get "decent" (with the expectations set right / low) coding performance out of Qwen3.5-9B on a decidedly medium end Radeon 9070 paired with a 5700x3d and 32GB of DDR4 RAM.

We can finally reason with and "talk" to our hardware.

The other implication here is that this is all software improvements and optimization. There might be a lot more wiggle room for improving quality over time. It seems the model and reasoning quality is improving faster than the hardware currently.

The over reasoning that Simon Willison highlights here is a real issue though. I've observed it with some of the OpenAI models as well. They are prone to overthinking and overengineering things.

What I would love is models that figure out their own appropriate reasoning effort given a task. I'm spending too much brain cycles worrying on what model speed, reasoning, and quality settings to pick. It's not just a cost concern it's also a time concern. Wasting a lot of time for simple UI tweaks because the model is set to high or ultra or whatever is counter productive. The last few iterations of frontier models seem to emphasize benchmarks and reasoning effort.

But of course the day to day reality of many developers is that they are trying to solve relatively simple problems compared to e.g. proving some so far unproven theorems, solving some Nobel prize level problems, etc. I'd love my tools to start making sane choices based on what I ask rather than defaulting to "boil the oceans". These tools need some kind of Auto select. Mostly Ultra is overkill and a waste of time and resources. And of course with local models, keeping simple things local is a nice option.

It's nice to have Sol Ultra extra fast as an option in my back pocket. But it's complete overkill 99% of the time. And it's not like most users make good choices here or are even capable of making good, informed choices. The models are more intelligent than the tool UX. Arguably, a local model of very modest size might be able to do better for this specific choice.

In this kind of moment, I really wished hardware manufacturing and demand situation is in much state. Imagine this can be accessible by everyday people with only 6 months hardware market gap. The societal impact would be much bigger.

Tried yesterday on my own laptop (a UltraCore 7 255H without dedicated GPU,with 32 GB RAM), it wasn't even starting thinking, even on a small context window (65k)

I wish we could have better hardware and I think the tech is there for a few years already.

I've gone in (too many) details last night with the calcs: https://news.ycombinator.com/item?id=49324600

his 128gb Ram laptop is quite extreme

I thought the same thing, and I generally do a lot of animation in my work, and the results in motion graphics with Qwen are impressive, I really fell in love with it

The recent 'Stolen Thoughts'[1] paper shows many excerpts of private reasoning for frontier models.

For a complex maths problem, Sol reasoned in 367 tokens before working:

  We need solve. Need interpret no person sits next to two other people = among binary string length16 weight8, no occupied chair whose both neighbors occupied, equivalently ab 111 substring. Endpoints cannot have two neighbors anyway; only avoid 111. Count binary strings length16 weight8 avoiding 111. Need N mod1000. Compute stepwise perhaps runs of 1 length max2. Count via runs.
  
  [... cut in half for HN readability ... ]

  Check interpretation potentially "no person sits next to two other people": no seated person's chair adjacent to two occupied chairs. Exactly no three consecutive chairs selected. yes.
  
  Need reason step by step final boxed 907. Explain runs. Ensure people each select chair distinct subset (given subset count). Let's present.

I think one of the bigger problems is that verbose, `max`-style thinking does generally lead to higher benchmark scores. And model vendors are incentivised to for benchmarks (at least to some extent).

[1] https://stolen-thoughts.com/

For a model this small it’s also a way of trading supply-constrained VRAM for inference time, which for self hosting consumers (and probably Chinese companies subject to export controls) is mostly good.

I can imagine a curve where for a given level of “intelligence” you either need model size, or inference time (“test time compute”), and can somewhat trade one for the other.

I believe this is what Meta is doing as they started recording their SWE screens some time ago.

For coding, this is very interesting because the same incentives were present for humans before AI. Tech companies which had a culture of rewarding complexity would see huge Pull Requests and a lot of unnecessary complexity. I've worked in companies which would require a thousand lines of code to implement a feature which would require only a hundred or so lines at a different company. The shorter one was more reliable too. Code begets more code. The incentives created by the company culture had a massive impact... And the culture was heavily determined by whether or not the company had a market monopoly. More monopoly power -> more unnecessary complexity (presumably so that employees could achieve better lock-in/job security through the increased need to manage that complexity; in any case, the company could comfortably afford and it did not present an existential risk as it would in a startup environment).

So it's not surprising that the same dynamics are at play with AI. Now, because code is being churned out so rapidly, the effects have become much more obvious (it took me years to figure this out, but now managers can observe this same effect play out in months); many senior engineers and CTOs will echo my point; but I suspect most engineers and crucially, most managers, still don't get it...

Something tells me that the AI companies supplying the models are well aware of the tradeoff. When you can dial up the complexity of the LLM's output by 5% (I.e. 5% more tokens to solve the same problem) and see a 5% immediate increase in your revenue from a large segment of your users, that's a very tempting knob to dial up! Now when you learn that this complexity compounds and next year's revenue will be 10% higher (purely as a result of your users now having to maintain that additional complexity); this is extremely tempting! Especially in the context of users who are largely ignorant as to the true cost of the unnecessary complexity they are adding... The insider's term for this is 'technical debt' for multiple reasons; including the fact that it compounds like normal debt. Now factor in the monopolistic tendencies of those tech markets... It's a real bottomless gold mine.

Revenue from downstream corporate users comes in regardless of compounding code complexity and slower pace of delivery; those companies just keep hiring more people, spending more on tokens. Swallowing up these massive diminishing returns like an appetizer. Enshittification takes place but the downstream end user has nowhere else to go.

It's trivial for a lab to advertise themselves as being token-efficient and almost impossible for its corporate users to actually verify it.

The only real issue with that business model (possibly a fatal issue) are these open weights models which the big tech companies could use to move off the AI service platforms if the problem becomes bad enough.

Is that similar to what ggerganov is talking about here?

https://x.com/ggerganov/status/2089214161884414147

What’s that about the template breaking the kv cache?

Unlike cloud infra in general which offers things like automatic backups, regional redundancy, and effectively unlimited scalability, it seems like the value proposition of cloud LLM gets ever shakier.

* Many businesses don't need frontier level intelligence anyway.

* It's completely stateless. If your local LLM machine catches fire? Nothing was lost. Buy another.

One line of evolution seems to be toward some form of latent-space reasoning, as in [1]. Natural language seems like a relatively low-bandwidth channel for intermediate reasoning.

[1] https://github.com/sapientinc/HRM-Text

It needs to argue with itself to extract most of the knowledge embedded in the weights into the context. Asking it to synthesize ideas directly in a single go is simply unreasonable. And MoE models need to walk multiple experts to extract all the knowledge on top of that. So you need to give them the reasoning trace to first spill all the associations into.

It “thinks out loud” to populate its token space. Asking it to shortcut risks truncating that process.

I was pretty happy with Depseek Pro in Opencode util I discovered I can see the thinking trace by clicking on the "thinking..." communicate. All those seemingly unnecessary "but wait" messages are frustrating to read. But at least to some extent it's just model taking time thinking through the problem, and the trace produced doesn't have to be representative of what happens internally: https://arxiv.org/abs/2404.15758

So-called "caveman" thinking attempts to address this.

The important part of "actually wait, I really need to XYZ" is just "XYZ".

The model can attend to just "do XYZ" and produce almost the same vector modifications as full verbose "reasoning".

I hated it at first too...

Now though I'm considering all the hidden "thinking" in the models layers that happens for each token output. It is a wild amount of waste! We just can't see it.

This kind of stupid excessive computation is fundamentally how these models are so good.

One day hopefully not so soon someone smart or a foundation model will come up with a more efficient architecture. That's when things get really scary.

Hardcode their "thoughts" in your agents.md... But they might still reason through it anyway.

Chain-of-thought output shouldn't be taken literally. The tokens are a substrate for computation, not necessarily evidence that the model is wasting time and electricity by gratuitously second-guessing itself over and over.

You can see evidence of this phenomenon in models dating back to the OG Deepseek R1. It was common to see the model talk itself out of the correct solution in the <thinking> block, or fail to reach it at all, only to produce a correct answer in the response. And vice versa; it was also common to see it reason its way to the right answer and then fail to follow through in the response.

For day to day LLM experimentation (and even some business use cases), I'd say Apple Silicon would be first choice for me.

Have you looked at using oMLX?

https://omlx.ai/

My impression is that when you allow the model to use internal thinking as opposed to asking it to output its thinking first, it's more likely to backtrack when I detects a flaw in its plan. Said otherwise: once producing user-facing output it seems to lock into an approach, for better or worse.

Yes, if you set reasoning to none you can force the granularity of the thinking.

It will actually adhere to your request for e.g. 3 sentences max.

Thinking mode will override any instructions in the prompt (at least for other models in my experience).

Of course this will probably hurt performance, but works great for easy tasks that you know are trivial. Tons of pipeline, image recognition etc use cases where this works well.

I'd be curious to see Qwen 3.8 27B low thinking benchmarks though.

I feel that local models are better for "processes" where you need a degree of predictability. Like summarising the daily weather for the family chat bot or analysing email inbox priority.

SOTA cloud models are more for open-ended tasks where you need "creativity".

To be fair a lot of models have quirks, I've never found a model swap that was transparent.

I'm doing much the same, avoid the long thinking loops and instead have more iterations on the plan with reviews from different angles.

Comparing with Muse Glimmer is a good idea. I ran the same exact HTML tool generating prompt against both Glimmer 30B and Qwen 3.8 27B. Results:

Qwen: https://gist.github.com/simonw/121ad098860028b2fab603fa12da1... - 17,576 reasoning tokens, produced this HTML result: https://static.simonwillison.net/static/2026/qwen-over-think...

Glimmer: https://gist.github.com/simonw/51e8ddb2ee597a5005fa63bd4927d... 1,021 reasoning tokens, this HTML: https://static.simonwillison.net/static/2026/glimmer-bbox.ht... - ugly but functional.

In both cases paste in the URL https://static.simonwillison.net/static/2026/two-pelicans-on... to see them work.

Both applications work correctly and fulfill the requirements. The Qwen one (which used the default xhigh reasoning setting) is massively over-engineered. The Glimmer one used whatever their default in LM Studio is and I would argue is a tiny bit under-engineered.

Weirdly the Glimmer one doesn't work with images on other domains like https://static.inaturalist.org/photos/714731804/large.jpg - it fails with a CORS error, but you don't need CORS to load images and detect their width and height, and the Qwen one handles that URL just fine.

That's because Glimmer added this unnecessary line:

  img.crossOrigin = 'anonymous';

I love reading Glimmer's "thoughts". Why use many word when few do trick?

Glimmer is stupider than 3.6 27B. You can't compare its speed to 3.8 and be done.

I am eyeing one of these specifically for this use case, could you please post roughly what kind of tokens per second numbers you get for text generation for this 27B model?

edit: and which quant you are using, please :-)

10 hours at what tokens per sec?

Was that with the default xhigh reasoning setting? I suggest trying again with reasoning set to low or turned off entirely.

xhigh tells it to overthink and re check everything. Low tells it to only do the minimum thinking necessary. I would suggest to give qwen medium which doesn't inject any thinking directives into it and also to give as much context as you can, ideally around 500k tokens or even 1M if you can. Big complex tasks like these make the model hit the compaction trigger a lot and they end up re thinking the same thing several times in my experience.

Opus 4.6 performance with a local model that can be hosted on consumer hardware is an incredible result!!

$0.286/kWh is a ridiculous amount of money to pay for power. That's more than double the regional residental average here!

If I ever found myself in this situation I would much rather just rent cards from hotasile and run open models instead of giving OAI money and playing reset bingo

Btw, unrelated, but this is the kind of Vibe Coded AI slop design I see a lot these days. Every single thing on this page has a different color, formatting, and it's just painful to look at.

I think the point is that a larger model with far less reasoning time solves the problem just fine. Which of course is the tradeoff: The smaller the model, the more reasoning you need to get decent answers to tough questions.

Agreed. Test driving the bad default is what they deserve (they brought this upon themselves as a benchmaxx attempt), but comparing it to running with reasoning completely disabled is also weird.

The word "open" has lost all meaning

The model itself is excellent, the defaults are bad. As Simon and other pointed out, medium is great.

Reminds me of Gemma4 and the official (or at least popularly used around launch) Jinja templates being wrong and broken for tool calling.

Yes. Though the chat template doesn’t tell LM Studio to offer the little dropdown. You can bodge the template in the load parameters.

Unsloth Studio / Desktop has it working really well with their version of the weights.

The feedback loops are getting tighter every day.

You need ~24-26GB for basic setup (17-19GB model + 128K 8 bit context), so you can, but not much memory would be left for doing anything else on that machine. And even then it would run at like 5-10 t/s (due to relatively low memory bandwidth of M3 Pro) and slow prompt processing (couple hundreds of t/s?)

If they end up releasing updated 35B-A3B variant, then it would be much more interesting in generation speed (~50 t/s)

For inference engine/server you have two (main) choices: llama.cpp for platform-agnostic, MLX for Apple-only. They will spin up OpenAI-compatible local server, and you point your agent harness to it.

For llama.cpp this should be reasonable (maybe shrink context to 128K) starting point: https://x.com/ggerganov/status/2088312671196082312

Depends on how you work with it. Reading the meandering and repetitive thinking is disturbing and taxing, so we can’t do that. So thinking has to be (mostly) hidden and just becomes waiting time.

Yes, for coding, they aren't overthinking enough. I want much more thinking and less code in the PR! Even with the best frontier models, I still have to guide them towards the right solution. The more thinking they do, the less code they write.

I have quite a complex codebase where I made a lot of nuanced decisions with regards to keeping the processes embarrassingly parallel, DB indexing, caching, async/await, backpressure monitoring, spam prevention, schema validation, etc... and now the agents are really good at adding features on top and prompting is minimal.

If you have a relatively large codebase and never even once cut a corner, then the AI agents tend to follow through with that style and the ratio of reasoning-to-code increases. Worth it.

Unfortunately in xhigh thinking it goes down rabbit holes in such an extreme depth-first way, that whenever you choose to cut it off, there is a very good chance it will not have got round to musing on even half of the prompt! It doesn’t really obviously loop in xhigh, so I am not sure if an “overthinking guard” proxy would have much to go on, but it does obsessively ruminate on edge cases. I have seen it overcomplicate simple code as a result even in my limited testing.

Probably the better solution if you want it to be quicker but still fairly thorough appears to be to configure reasoning effort instead of thinking budget. It seems to do very well still even on the Low setting; on the Medium setting it can get stuck in loops like 3.6 does.

I think xhigh reasoning effort was an absurd choice for a default, and so was not sorting out the chat template so LM Studio could offer the reasoning effort dropdown.

Yeah, but be fair. Working with small models is a different ball game. Not all the batteries come included :)

This is definitely such a cool feature that I wish cloud providers would expose.

I think that in this case there is also the problem of trying to transfer MoE-style reasoning into a dense model. I mean, MoE needs reasoning to walk multiple experts, but dense model already has all the weights. So when you push it hard to reproduce the MoE traces, you are effectively asking a small mouse to role-play as an anthill. Not great.

Checkout MTPLX. I use the Youssofal/Qwen3.8-27B-MTPLX-Optimized-Speed model and I get around 12 tok/s on a 36GB MBP.

AFAIK that is initially only for the iPhone?

You mean this?

  draw an svg of a circle

This morning I tried experimenting with this ThinkingCap lora I found someone made for 3.6 https://huggingface.co/signsur4739379373/Qwen3.6-27B-Thinkin...

ThinkingCap is a 3.6 27b finetune that claims to halve thinking tokens while maintaining the same output quality. I've used the model a lot and I'd say it holds up. Since 3.6 has the same architecture as 3.8, the lora can be applied.

With the prompt "create a fancy circle in html", these are the results for xhigh, medium, low and xhigh + thinkingcap lora

https://gist.github.com/CapsAdmin/b0ea64006f942c5a96a56dba78...

(Note that the gists are bloated because they contain the full chat and launch params in text/plain script tags for transparency)

I'd say xhigh looks a little better than xhigh + lora, but the lora variant has 40% less thinking tokens. Both seemed to take the same approach with adding random details that weren't explicitly specified.

Medium and low (no lora) are close to each other but are much simpler results.

This is just me testing a single turn. I haven't tested this on multi turns and whatnot, but I thought the result was interesting enough to share anyway.

I had to fix this on 35B A3B -- I have a proxy that just shuts it down if it gets to 2K thinking tokens and injects something like "We have thought enough, let's begin working." and it almost always finishes the turn then. It rarely needs more than 2K thinking tokens and if it does there is always next turn. I would need to see what 27B is actually doing, but these smaller Qwen models seem prone to this.