Thread

Bonsai 2 27B: Near-Lossless Compression in a 9x Smaller Footprint

589 points · 200 comments · JonSchneider

adrian17 · 1789685186 · focus · HN ↗
> Ternary Bonsai 2 27B uses ternary {−1, 0, +1} weights with FP16 group-wise scaling, for 1.76 effective bits per weight
If I recall correctly, a recent post [1] has shown that Q2 quants (with like 2.6 bpw) of the same base Qwen model sit at the edge between "noticeably worse" and Q1's "useless". I took a quick glance at Bonsai's blog posts, and don't really see them comparing themselves to "typical" quants or explaining what's the special sauce that makes them better?
<a href="https://news.ycombinator.com/item?id=49611128">https://news.ycombinator.com/item?id=49611128
1. edflsafoiewq · 1789685632 · focus · HN ↗
  I think the general idea is naive quantization falls apart below 4bpw but you can go lower with more sophisticated QAT-adjacent methods. Bonsai's quantization method is proprietary though.
  1. yowlingcat · 1789689412 · focus · HN ↗
    That's correct. I think there can certainly be issues even with 4bpw with naive quantization (IE you'll notice far better results from a QAT 4bpw vs a naive 4bpw).
    One such method that I've been meaning to look into further is Tencent's AngelSlim QAT/PTQ approach. They did a Hy4 preview release thats an STQ_1_0 at 2.38 bpw:
    <a href="https://huggingface.co/AngelSlim/Hy4-preview-GGUF" rel="nofollow">https://huggingface.co/AngelSlim/Hy4-preview-GGUF <a href="https://arxiv.org/abs/2602.21233" rel="nofollow">https://arxiv.org/abs/2602.21233
    Of course, it's still 213g of VRAM I'd need so it's somewhat out of the range of what I can run locally. In contrast, this new Bonsai is nice because the original was already exciting for making use of low VRAM devices. Could breath new life into some of the older GPUs that were previously close to top of the line just quite VRAM constrained by modern standards and still quite cost effective for now.
  2. om8 · 1789692235 · focus · HN ↗
    Could've been better if GGUF implemented QTIP format. GGUF representation is a major limitation for llama.cpp quantization performance
    edflsafoiewq · 1789692534 · focus · HN ↗
    They use their own llama fork anyway, so that shouldn't matter.

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.