Thread

1. infogulch · 1789600676 · focus · HN ↗

   So they get down from 1.58 to 1.48 bits per weight by exploiting the fact that actual weights in practice are 0 51% of the time. Neat.

   If ternary llms work out and are baked into hardware as custom silicon I bet they'll be shockingly efficient.

   1. kadushka · 1789601079 · focus · HN ↗

      By “work out” you mean no accuracy degradation? That’s a big ask - currently we can barely quantize to dynamic fp4 with small block size - still not completely lossless on all benchmarks.

      1. danielmarkbruce · 1789606173 · focus · HN ↗

         You are conflating post training quantization and low bit training.

         1. kadushka · 1789609334 · focus · HN ↗

            That's what I meant - we are currently use fp4 formats for training, and we cannot quite get away with that, despite dynamic quant and small block size - we still have to use quite a bit of higher precision (fp8 or even fp16) in various model components.

            1. danielmarkbruce · 1789618434 · focus · HN ↗

               I might still be misunderstanding what you are saying, but bitnet also keeps high precision latent weights during training. The optimizer updates those, while the weights used in the forward pass are quantized to ternary values.

Unofficial Hacker News client; not affiliated with Y Combinator.