Because streaming only removes the decoder stack. The embeddings and lm_head stay
resident, that is 2.10 GB of the 3.32 GB peak on 8B. And the logits tensor scales
with batch x seq x vocab, not with depth.
So it goes from "whole model must fit" to "embeddings plus one layer plus logits
must fit". That is why 8B works and why I did not try 14B.
The table on the site is the normal resident path, streaming is opt-in and BETA.
Should be clearer, my fault.
ranger_danger · · focus · HN ↗
MakazhanAlpamys · · focus · HN ↗
So it goes from "whole model must fit" to "embeddings plus one layer plus logits must fit". That is why 8B works and why I did not try 14B.
The table on the site is the normal resident path, streaming is opt-in and BETA. Should be clearer, my fault.