Thread

1. om8 · 1789598435 · focus · HN ↗

   Ternary quantization does not make any sense. Vector quantization and trellis based methods are better in this region for PTQ.

   1. janalsncm · 1789600366 · focus · HN ↗

      PTQ and vector quantization aren’t used for this because part of the point of ternary LLMs is to make them faster. In a ternary LLM every weight is an add, subtract, or no-op so it is fast on CPU.

      If you’re just using a code book to reconstruct a f16 model the only savings you can get are in sending it over the wire.

      1. mitxela · 1789601737 · focus · HN ↗

         which is important though since sending it across the wire over and over and over is actually the main bottleneck.

         1. Kerbonut · 1789625349 · focus · HN ↗

            Wire typically means internet connection, and it’s hardly the bottleneck

            1. [deleted] · 1789626642 · focus · HN ↗

               [deleted]

            2. 317070 · 1789629285 · focus · HN ↗

               in the case of large language models, the wire is the communication of your parameters between your layers of memory that is often the bottleneck. To do a forward pass, you need to use all parameters once, and so the communication between the compute and the storage is the bottleneck, and that bottleneck is also a bunch of wires.

               1. mitxela · 1789637008 · focus · HN ↗

                  The other bottleneck is the amount of fast storage, which compression also improves.
      2. om8 · 1789609201 · focus · HN ↗

         > If you’re just using a code book to reconstruct a f16 model the only savings you can get are in sending it over the wire.

         That’s why you need to use efficient gemm kernels like FLUTE for inference. They are ~as good as what you can do with ternary quantization.

      3. WithinReason · 1789626975 · focus · HN ↗

         And storing it in memory. Memory is expensive.

Unofficial Hacker News client; not affiliated with Y Combinator.