Thread

Bonsai 2 27B: Near-Lossless Compression in a 9x Smaller Footprint

589 points · 200 comments · JonSchneider

danbrooks · 1789684929 · focus · HN ↗
Nice! Does anyone know how this compares to the Unsloth quantizations of this model? <a href="https://unsloth.ai/docs/models/qwen3.8#run-qwen3.8-guide">https://unsloth.ai/docs/models/qwen3.8#run-qwen3.8-guide
1. 0xbadcafebee · 1789689422 · focus · HN ↗
  Came to ask the same. From my really rough understanding, it seems like Unsloth's method allows a slightly higher precision at a higher file size, while PrismML's uses a different approach to achieve a smaller size (and presumably less precision).
2. nulld3v · 1789693465 · focus · HN ↗
  There's a table on the HF page that compares it against Unsloth's UD-Q4_K_XL and IQ2_XXS: <a href="https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf#full-per-benchmark-results" rel="nofollow">https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf#fu...
  1. kadoban · 1789698822 · focus · HN ↗
    Oh, wow, they think it's just a smidge below the q4? That's crazy good if true.
    anana_ · 1789700591 · focus · HN ↗
    The benchmarks they chose are rather cherry picked to not include long context or difficult ones that involve long horizon work or many agent turns, as I suspect this is where the model shows more differences compared to the full fat one
    SillyUsername · 1789712098 · focus · HN ↗
    Yep more hops from the lower Q is likely going to skew the vectors further over time.
    I wonder if there's a way to mitigate this by running it through an original Q8 draft model, attuned somehow for the PTQ1 quant, but giving it a higher threshold for the acceptance linear with the context length itself?
    The longer the context, the higher the multiplier on the threshold, and more likely the draft result is used. Not ideal but it may extend the usable max context.
    This model might, even without this, be amazing for short lived agents that work via generations / have changing tasks.
3. [deleted] · 1789698305 · focus · HN ↗
  [deleted]
4. WithinReason · 1789713854 · focus · HN ↗
  Unsloth has been dethroned by ISTA:
  <a href="https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF" rel="nofollow">https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF
  The 3-bit quant is lossless based on benchmarks.
  1. raylad · 1789725475 · focus · HN ↗
    I just checked that model (ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF:IQ3_S) and it does much much worse on the "Please recite Jabberwocky" test than the original bf16 does.
    The bf16 only misses "snicker-snack" and this quantization becomes confused after the first stanza.
    sorenjan · 1789734089 · focus · HN ↗
    I think it should be expected that a smaller model is worse at reciting memorized data than a big one. I also don't think it's a good use case of small local models. Can it find and recite Jabberwocky if given access to a web search tool?
    0xbadcafebee · 1789755878 · focus · HN ↗
    Forgetting things isn't lossless though is it? Makes the benchmark and the finding quite suspect
    sorenjan · 1789828554 · focus · HN ↗
    It depends on what you mean by lossless. If both models can perform the same tasks it can be considered lossless for those tasks. That task might be more related to language understanding rather than memorizing, they have several benchmarks in the article.
    WithinReason · 1789737626 · focus · HN ↗
    I tried and ended up with:
    I'm going to stop here and be direct: I'm having trouble recalling the exact text, and every attempt above is me guessing. Rather than present a mangled version as the real poem, I'd recommend you look it up — it's very short and in the public domain, so any text of Through the Looking-Glass will have it verbatim. If you'd like, I can help with the moral of the poem ("'twas the blessing of the Bird..."), the famous Humpty Dumpty word interpretations ("slithy" = lithe + sinister, "mimsy" = miserable + mys... etc.), or Carroll's original annotations for the coined words — that part I can do reliably.

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.