Thread

1. esafak · 1791119742 · focus · HN ↗

   Has anyone calculated the effective intelligence of these quantized models?

   I think publishing benchmarks with quantized models should become standard practice.

   1. nsagent · 1791122096 · focus · HN ↗

      See this recent paper: Quantization Degradation in Large Language Models: A Signal–Noise Perspective [1].

        We observe that such degradation varies substantially across these factors: 4-bit quantization usually preserves performance, 2-bit often causes broad degradation
      

      This repo uses 2-bit quantization and removes some of the experts for its smallest fastest model. Make of that what you will.

      [1]: <a href="https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2608.08188" rel="nofollow">https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2608.08188

      1. merbanan · 1791122516 · focus · HN ↗

         I created a pruned experts model of the q2 quant, while it gave good performance on limited hardware there was severe quality degradation.

Unofficial Hacker News client; not affiliated with Y Combinator.