Thread

1. abraxas · 1789680452 · focus · HN ↗

   I'm not following the local mdoel scene too closely but this seems quite amazing. Is this able to be run on Apple silicon too?

   1. kamranjon · 1789680506 · focus · HN ↗

      "Ternary Bonsai 2 27B reaches up to 143 tokens/second on NVIDIA GeForce RTX 5090 and 46.8 tokens/second on M5 Max. On an RTX 4090, Ternary Bonsai 2 27B consumes just 0.714 mWh/token, making it 40% more energy-efficient than an 8B model running in full-precision."

      1. pizza234 · 1789681997 · focus · HN ↗

         Their mention of the 5090 is bit odd, since on 32 GB GPUs, Q6 fits while having better quality. Very interesting model for 16 GB GPUs though!

         1. pwython · 1789704984 · focus · HN ↗

            Sometimes you want a decent model running in the background that doesn't take up all the VRAM.

            1. blurbleblurble · 1789705537 · focus · HN ↗

               Or maybe even to run parallel threads of the same model!

Unofficial Hacker News client; not affiliated with Y Combinator.