Hacker News

The nice thing is that current local modes don't go away. The weights are out. Qwen 27b and deepseek are already very useful. Hardware will keep getting stronger so after the current memory crisis you can run them more cheaply.

Of course the proprietary models will continue to improve and we have no guarantee that there will be new open models. But for coding, the local models are already good enough for a vast amount of use cases, if you are an experienced developer yourself and can guide it.

But it's all for the good of humanity, so how can you complain?

https://forum.effectivealtruism.org/posts/53Gc35vDLK2u5nBxP/...

>Anthropic restricting game reverse engineering

Which is literally the coolest thing to come out of this...

Did they get a letter from Nintendo's legal team or something?

Wait and see, I give it fairly high odds everything tagged as 'uncensored' or 'abliterated' will get wiped from huggingface at some point in the future. The main barrier to wider spread adoption of this right now is that very few people have the legit on-prem hardware resources/budget to run something like an abliterated Q8 of deepseek v4.1 flash 1017 or mimo 2.6 or equivalent in size locally.

I got banned from the Hermes discord today not gonna lie.

https://gnosh.ai/head-to-head.html

Did we ever get back to 16GB feasibility?

AFAIK the timeline was:

  - AI was shit, models could fit in your GPU
  - AI was shit, models could no longer fit in your GPU
  - AI got good

> because with the Bonsai 2 quantisations it is quite achievable to get Qwen 27B running at 50 t/s on reasonably priced gaming machine

The Bonsai quantizations are not good for real work. I have tried them. They are as bad as you would expect from 2-bit quants. They may have done a good job quantizing to that limit, but it’s still bad.

Even the 4-bit Qwen3.8 27B quants are still iffy for coding work. It’s not until Q5 or Q6 that I find them useful, which is too bad because that cuts into available context on 32GB cards.

> Which in turn suggests that Fable 5 coding performance can be achieved locally with a bit of clever harnessing.

I experiment with a lot of local inference engines and I also use both Fable and Astra. I wish this was true, but it’s just not. I can generate tens of millions of Qwen3.8 tokens per day and run the hardware nonstop, but even brute forcing it like this just can’t accomplish the same thing that I can get Astra, Fable, or even Opus and Sol to get done in an hour or less most of the time. You just cannot make up for the lack of intelligence built into the model by brute force unless maybe your timeline extends to infinity.

Bonsai 2-bit quantization is notably dumber than Qwen 27B at eight bits or even four bits. I doubt you'll get this level of quality even with five of them.

The bonsai quants are not that awesome. Yeah they're fast and they retain the "27B" but in practice the perform similarly

5GB of bonsai weights end up being similar to 5GB of QAT weights even when one is derived from a 27B and the other from an 8B

Bonsai 2 has a bit different architecture as far I understand and speculative decoding is not so great yet. Before that point, original model is faster if you have the memory. Would be nice to see the speed with proper speculative decoder.

Happy hallowen! ::::

Actually we tried Cerebras. It caps output at 32K tokens so the 128K reasoning (the author budget) didn't fit. Cerebras speed is real though

I was getting deja vu. Copy paste my comment from that thread to here I guess. Apparently it's really easy to get Internet points by making up a harness and showing that good open source models can do as well as the frontier, you know, on useless saturated and contaminated benchmarks from years ago that don't reflect any of the agentic work we actually use these models for. It's so stupid because these models are at the level of a decent non fable frontier model from 3-6 months ago and it's also cool that you could probably get equal perf to gpt 5.5 from qwen 3.9 27b or gpt 5.6 sol/opus 5 from a quantized DeepSeek or glm 5.3 flash. Those models were pretty useful in their time! Instead we have whatever this new form of harness/benchmark slop is

Here the intent is to replicate GVS5H with memdoor a headless coding agent I created months ago. Anyone can reshape the harness by redefining the steps in yaml files , models and temperatures. Up to you guys to try you have the recipe in this repo.

Well, Greg KH emphatically states that local LLMs are more than good enough for bug finding in the Linux Kernel. [0] Seems strong enough for the task to me. [1]

He goes on to recommend that one uses local LLMs exclusively for bug finding because he and his colleagues have seen multiple occasions where folks who used the LLMs provided by the Major LLM Manufacturers had the stuff they were working on get "mysteriously" leaked.

Even if that sort of thing were not known to happen, it's just smart to not process data you want to keep secret in systems that are known to straight up retain user data and/or "anonymize" user data and send it back to their HQ for use in production of newer versions of their products.

[0] IIRC, it is this video: <https://www.youtube.com/watch?v=NnV_cWeoo5Q>

[1] And even if they're six months behind the latest publicly-available tools provided by the Major LLM Manufacturers, six months ago a notable subset of folks were on here screaming that their latest tools enabled fully-automated hands-off coding and had ushered in an era of Software Factories that were rapidly going to make all programmers obsolete. ;)

I've built an RSS reader and I'm doing a half assed reverse engineer / web version of Tapped Out (a silly little Simpsons mobile game that EA killed a while back). I've had to go to Claude a couple times when it got stuck on something curly, but I think that was mostly just loading a fresh context and I could have solved it I'd I really tried.

I've been using Qwen 3.8 and more recently Next for the last two months almost exclusively on home projects (I have access to a system with an RTX 6000 for free). It does everything I want it to! I have no issues believing it would be fine for more than just pet projects.

Well in large corp this takes time, even if you have an army of Einsteins at your disposal (just like we saw with Elons Doge) because there will be 16 different spreedsheets/dbs talking abt the state of the system that all contradict each other. While the 6 people who maintain their own state to keep the lights on, will have job titles that have nothing to do with system state maintenance.

Small companies its hit or miss cuz they are overwhelmed by the pace of change. Basically no one wants to hear things will take time. Intelligence is just one system variable and others dont just auto sync.

This is a benchmark that solves ten hard problems in five passes. It scores 9.8 against the authors' 9.6 and Fable 5's 8.0. Memdoor workflows are the harness that replicates the original GVS5H thread harness posted in HN last week. Raw results (i.e. each call's tokens, cost and host) are in the bench folder for verification.

will definitively try it. Overall with a smarter model the improvement is in the token usage and latency. Tbh I think i could get a 10 score if i tuned the prompt and temperature a little more. The original author harness is already giving 9.6 score which is pretty good. However this harness loop can take more than 1 hour on the hardest problem at 100 token /sec. A smarter model such as 3.8-flash-next could get 10 score and resolve the hardest problem faster. That's my estimate i'll give it a try. Thx

I would not think in terms of "how much" context, but really just how the harness manages it. I've been fiddling around with harnesses earlier this year and the rabbit hole goes very deep. I eventually had to drop it because it consumed me, and I can only do one thing well. But what I learned is that there is a lot on the table at the harness level.

The elephant in the room and the constant tension is that the models do get "used" to a certain harness (trained for it by the model labs), but they also generalize well. So one really advanced mechanism could fall short because the model could fail to generalize, whereas in other cases the model might handle large deviations from how a conventional harness works better than expected. This is where the gains are.

"think", "believe" - maybe, but this is why we need better measurability.