Bonsai 27B (1-bit LLM): The First 27B-Class Model to Run on a Phone (prismml.com)
44 points by xenova an hour ago
liuliu 32 minutes ago
The problem, of course, is if you run the UD_Q2 variant (Unsloth) which does only post-training, the number is pretty close to 1-bit model here and the 5% drop in tool-call is significant than it suggests in real-life use cases.
liuliu 22 minutes ago
You also need to pay close attention to BFCLv3 multi-turn result, that helps you to get a sense how frequently these quants will be in a doom loop.
simonw 18 minutes ago
The models themselves are showing up on Hugging Face here: https://huggingface.co/prism-ml/models
xyzsparetimexyz 11 minutes ago
That's awesome. What's the largest model that could fit onto a single 16gb gpu at 1.125 effects bits per weight?
alvatech 39 minutes ago
TIL that 1 bit models are actually 1.58 bit with three values +1, 0 and -1
NitpickLawyer 17 minutes ago
There's two variants of this (or, as the joke goes, for very big values of bit):
Ternary Bonsai 27B uses ternary {−1, 0, +1} weights with FP16 group-wise scaling, giving a true 1.71 effective bits per weight.
1-bit Bonsai 27B uses binary {−1, +1} weights with the same group-wise scaling, giving 1.125 effective bits per weight.
bensyverson 29 minutes ago
Yeah, it's an unfortunate convention from the very first "1 bit" model. But to be clear, Bonsai comes in both ternary and actual 1-bit variants.
Havoc 19 minutes ago
This must be some sort of unpublished app?
I can just see their image tool on the app store