Different angle on the same model: the full GLM-5.3 (744B MoE, 4-bit experts, 434 GB on disk) runs on a single MacBook Pro M5 Max with 128 GB
by streaming the experts from NVMe SSDs instead of keeping them in memory.
One drive gives about 2 tok/s; striped across four drives it reaches 3.5 tok/s with byte-identical output, and our best internal build with a not-yet-published patch does 4.2.
Method and numbers: <a href="https://github.com/argonautlabsai/argodrive" rel="nofollow">https://github.com/argonautlabsai/argodrive (built on antirez/ds4).
Argonautlabs · · focus · HN ↗
One drive gives about 2 tok/s; striped across four drives it reaches 3.5 tok/s with byte-identical output, and our best internal build with a not-yet-published patch does 4.2.
Method and numbers: <a href="https://github.com/argonautlabsai/argodrive" rel="nofollow">https://github.com/argonautlabsai/argodrive (built on antirez/ds4).
tipsytoad · · focus · HN ↗
Argonautlabs · · focus · HN ↗
[dead]