Thread

1. Tepix · 1791122120 · focus · HN ↗

   Q2 quantization. Not interested.

   1. tcdent · 1791123592 · focus · HN ↗

      All of these projects targeting low spec systems and "100 tok/s" are the same 2 bit quant without much else. Conveniently none of them include any mention of accuracy in their published numbers. 4 bit is the floor.

      1. latentsea · 1791130879 · focus · HN ↗

         You can run a 4 bit quant with this. Personally, I switched to running IQ3_XXS and am getting better outputs than 27B and at faster speeds.

Unofficial Hacker News client; not affiliated with Y Combinator.