Thread

1. hedora · 1789691625 · focus · HN ↗

   RAM requirements? My current rule of thumb is “a byte per parameter”, but I doubt this runs in 1/9th that (~ 3GiB).

   Also, perf speedup?

   1. kennywinker · 1789710366 · focus · HN ↗

      Without context, it should be number of params * 1.76 (the “effective bits per weight”) / 8

      So for this one, 27B * 1.76 / 8 = 5.94 GB

      For speed, far as I can tell it depends if your gpu is memory bandwidth bound or (mostly older gpus) processing bound. If it’s memory bandwidth bound, and your gpu gets 300GB/s, that’s:

      300GB/s / 5.98 GB = 50.5t/s.

      Realistically it’s probably a bit slower, but that is your theoretical maximum.

      1. Dwedit · 1789741404 · focus · HN ↗

         It needs more VRAM than just the model weights. With 6GB of VRAM, I got 44/65 layers loaded into VRAM. Has anyone tested 8GB yet?

         1. kennywinker · 1789755914 · focus · HN ↗

            “Without context” was me gesturing at that. Interesting you can’t fit the whole model tho - why is beyond my current understanding :)

            1. Dwedit · 1789756700 · focus · HN ↗

               Even with a small context size (4096 at 64KB per token), that's like 256MB for the context. It's more than just the context that's eating up VRAM.

Unofficial Hacker News client; not affiliated with Y Combinator.