Hacker News

Point taken, but there is a much more fundamental issue with it - and precisely why I wrote "very conservative".

It is a different problem if we pick two sets from the same data distribution, A and B, and first we have a score on A, then on B. Here we re-run on precisely the same set of Terminal Bench 2.1 problems. It may be that results are so random between runs that each single task has the same probability in a Bernoulli distribution. But more likely, many problems are easy (i.e. each run will solve them consistently), many are too hard (i.e. no run is going to solve them) and only a fraction is somehow in between.

Maybe there is some good trick to find a proper distribution, but to my knowledge, we would need to run it at least two times on TB2.1 to get any more educated estimates. That said, I am open to new ideas.

That said, I consider frequentist probability a dirty trick, and that Bayesian is the proper way of doing things (vide David J.C. MacKay" Information Theory, Inference, and Learning Algorithms" and Cam Davidson-Pilon "Probabilistic Programming & Bayesian Methods for Hackers" https://www.inference.org.uk/itprnn/book.pdf, https://dataorigami.net/Probabilistic-Programming-and-Bayesi...).

> The key confusion underlying the FCF is the confusion of what is known before observing the data — that the CI, whatever it will be, has a fixed chance of containing the true value — with what is known after observing the data. Frequentist CI theory says nothing at all about the probability that a particular, observed confidence interval contains the true value; it is either 0 (if the interval does not contain the parameter) or 1 (if the interval does contain the true value).

They're really reaching here. There's a 95% chance it's 1 and 5% it's 0, and we don't know which, so it's 0.95. Like saying the probability your uncle died of a shark attack is either 0 or 1. But in reality I don't know anything about your uncle so it's the average probability that someone picked randomly died of a shark attack, until I know it about your uncle.

Task success is not enough; reliability and cost matter.

Yah, prediction interval instead, right? (I'm still learning statistics)

Saying there's a confidence interval for run-to-run makes no sense, from the way I understand it, because each run could have a result that's all over the place.

Mind blown. The more I read about statistics, the less I know.

Thank you for these, coz I learned a lot! Great that they are open access.

the 2nd derivate of shannon covariance of noise begs to differ

There was a study specifically related to Qwen3.8 27B that showed that kv cache quantization has almost no impact on this model all the way to q4:

https://arxiv.org/html/2609.04098

On many models that I tested in past context quantization had very bad effect on model performance. However qwen3.8 27b is different.

I'm now running NVFP4 quantized both weight and cache on my RTX5090 and getting excellent results: 264k cache allocated for pool, 10k tok/s prompt processing, 200 tok/s generation for single stream, or 801 tok/s generation for 8 concurrent streams. Also have about 2Gb vram left for use of OS.

my coding agents regularly reach 200k context used without noticeable degradation.

P.S. I used setup from: https://github.com/seanyourhighness/vllm-sm12x-nvfp4-dflash2

You don't have to quantize all layers and all dimensions uniformly, FWIW

Not for me. As stated elsewhere, even Q5 (!) seems to be enough to kill the model’s ability to solve a particular problem in reasonable time. But that might just be right at the edge of what the model can do in the first place.

I have another personal benchmark problem (of a very different nature) that Qwen3.8-27B usually can’t solve at all, while Opus and GLM-5.3-Flash solve it consistently and very beautifully.

As they say, time is money.

In the age of the rampocalypse, the peasants may not have a choice between the two.. time it is!

I personally think thinking is basically variable but rate precision. If you are in a 4bit mode but need 2x as many tokens you're just doing fp8 with hoops( of course 4bit multiply is faster)

If it digs itself into a hole, try low or medium. In the rust coding benchmarks (on my machine) it did better on low and medium because xhigh never finished.

Running Q3 on my AMD RX 9070XT. 32k context and 32/TPS. Apart from the context window preventing it from doing any large tasks, this thing is seriously powerful. I could probably push it to 64k context. Local open models are the future, and I am definitely getting a more powerful card. Very fun!

There is an interesting new dynamic 3 bit quantisation I have been meaning to test:

https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF

Luke of Luke’s Dev Lab on YouTube had a look at it. It seems to outperform the typical 3-bit quantisation but whether it outperforms the new Unsloth dynamic I don’t know.

Q3 XL and Q3 XS are the two I'm trying to decide on

I have a 5080, three OpenAI Pro token resets, and I’m on paternity leave. Astra seems pretty clever. Maybe I’ll give it a task.

Before this experiment I tried to run KLD on various context lengths to see a quantization-dependent deterioration.

On wikitext2 there was no difference. I concluded these have no long-term dependency and I should use Linux kernel. Still, the same.

So yes, KLD depends on the dataset. Still, it does not measure what any e2e test does.

> A common issue is that it's rarely mentioned on which dataset KL-divergence is computed. It seems the most common dataset is wikitext

Thank you for calling this out. Using Wikipedia snippets for these is a terrible choice. I did a bunch of KL and other stats with the five Gemma 4 models, and the results were non-obvious. Anthropomorphizing:

Gemma 4 31B: "I guess we'll pretend I said this, but it's not me." (Baseline for stats)

Gemma 4 26B: "Dude, I'm certain I wouldn't have said this." (Bad KL)

Gemma 4 12B: "Umm, Me either!" (Similarly Bad KL)

Gemma 4 E4B: "I might say almost anything, this is fine." (Much better KL!!!)

Gemma 4 E2B: "I'm basically a toy. Let's play a game!" (Same KL as E4B)

Anyway, for comparing quantizations, it seems like the largest precision version should be given a one-shot prompt, and the result from that should be used as the corpus for the quantized versions.

I wrote this blog post myself, with AI for proofreading (typos and grammar, but not style). There were a few singular sentences for which I had a writer's block, but not much besides that.

So, if there are irrelevant remarks, these are mine. :)

Charts are vibe-coded - but it took quite a bit of hand-holding to get something decent. And the logarithmic scale for model size is my conscious choice (against Claude's initial ideas).

IMO, whether or not an LLM was used in the writing process doesn't really matter and I think it's a bit annoying that articles are being dismissed out of hand because of that.

The line is "Is this an interesting and accurate article that concisely makes it's case".

LLMs love to burn paragraphs writing about nothing which is why it's generally poor writing. Humans can do the same thing if they are trying to make very little information feel more substantial.

I say, stop trying to determine if an LLM was used and start judging based on your subjective measure that you'd have used before LLMs became widespread.

Just ask your freaking agent to read it for you and extract the information. That's what it's all about. Why would I be reading these articles other than information?

I think the advice is the same regardless of AI use: read articles written by authors that have a history of high quality writing.

You don’t need anyone’s permission. You have only so much attention, why spend it wading through slop?

I thought the wisdom is more so don’t bother going below 4bit and you won’t see a difference above 8bit.

Those are multiple-choice questions. If some of them are "trick questions", where obvious answer (e.g. the value taken directly from question's text) is wrong, bad model might perform worse than a dice.

On the other hand, not sure where from 25% baseline for random answers come from. Since this is multiple-choice-out-of-4 test, random guessing should be correct in 1 in 15 cases, not 1 in 4.

Yes, it's fine to run a model on bare metal. The model is just a token predictor. Leave out the fine semantics about this; it's a function taking a set of input tokens to output tokens.

It can't mess with your computer or files until you hook it to a harness, which interprets some of the model's output as commands to execute. So, model on bare metal, harness in a container or VM.

It looks like they tested Q4_K_M which should be just the standard K-quant without any imatrix calibration. The smaller ones are indeed dynamic though.

> Qwen3.8 27B seems like it was clearly supposed to be a high-end consumer open-weights model, but the t/s is so low for me on my old M1 Max 64GB that I hope others are getting use out of it.

Have you tried it with MTPLX? I get around 30 tok/s with it, also on an M1 Max with 64GB.

I leave it running at night. No danger of burning my token subscriptions and it has hours and hours to run slowly with a manager like: github.com/kunchenguid/gnhf

Have you tried https://huggingface.co/prism-ml/Bonsai-27B-mlx-1bit ? PrismML is the only people i am aware of doing 1bit that is decent.

I've ran some agentic stuff with Qwen3.8-27B-UD-Q4_K_M on my RTX 3090. It's fast enough to be usable in my opinion. But Qwen3.6-35B-A3B in the same quantization is much faster

I treat it like image gen. Send a prompt then come back in 40 minutes.

Too late to edit, but a few other things to note: I minmaxed the draft config. On my typical coding workloads it gets around 70% acceptance, more variable on prose.

The chat template is froggeric's fixed qwen template, v22.5 as of today.

It literally is usable now. A 5060 for $800 can run qwen3.8-27b 4bit at >40t/s, and the model beats opus 4.6 (max).

In my own experience, qwen3.8-27b 4bit can consistently find bugs in software written by sonnet 5 and opus 5. But it does do that at maybe 1/10th the speed. Still a pretty good deal if you're coding without wanting to spend big.

qwen3.8-27b 4bit has a following specifically for being exceptionally gifted for such a small model.