Thread

1. syntaxing · 1790023047 · focus · HN ↗

   All these new models are such tease for us folks with 128GB of shared memory. Buying another unit now to expand to 256GB is a mortgage payment but it’s getting tempting…

   1. brcmthrowaway · 1790023227 · focus · HN ↗

      Is there a gamechanger around the corner to reduce DRAM requirements?

      1. stymaar · 1790023551 · focus · HN ↗

         n-gram per-layer embeddings[1][2] might be it.

         [1] <a href="https:&#x2F;&#x2F;sebastianraschka.com&#x2F;llm-architecture-gallery&#x2F;per-layer-embeddings&#x2F;" rel="nofollow">https:&#x2F;&#x2F;sebastianraschka.com&#x2F;llm-architecture-gallery&#x2F;per-la...

         [2]: See DS 4.1-Flash and Qwen-3.8-Next.

         1. verdverm · 1790023718 · focus · HN ↗

            this is to offload VRAM to DRAM (for GP comment), and makes no difference for URAM

            1. petu · 1790025017 · focus · HN ↗

               n-grams can be kept on SSD, no need to hold them in any kind of RAM (at least w&#x2F;o batching)

Unofficial Hacker News client; not affiliated with Y Combinator.