‹ BackHN Continuity

Thread

Bonsai 2 27B: Near-Lossless Compression in a 9x Smaller Footprint

589 points · 200 comments · JonSchneider

  1. jedbrooke · · focus · HN ↗
    Running at about 7-8 tok/s (~60 tok/s prefill) on a Mac Mini M2 16GB.

    So far feels smarter than Bonsai 1 27B, it’s slightly larger than the Q1_0 quant. Super exciting stuff :)

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.