Thread

1. dghlsakjg · 1785816328 · focus · HN ↗

   I know everyone wants to crap all over these setups that are impractical, but this is how progress happens.

   People will keep plugging away at this and figure out how to avoid wearing the hard drive, how to make it run faster, custom hardware buses etc.

   Keep going! I personally can't wait for the day when a 1t param model runs off a $200 SSD instead of a $50k rack of Nvidia chips.

   1. pizza234 · 1785836695 · focus · HN ↗

      > but this is how progress happens.

      This is progress in the same way that a man climbing a tree is making progress toward reaching the moon.

      This project is essentially the MoE-of-the day, with some platform-related optimizations.

      > Keep going! I personally can't wait for the day when a 1t param model runs off a $200 SSD instead of a $50k rack of Nvidia chips.

      That won't happen. Projects like this just give the illusion that that will be possible.

      1. rhdunn · 1785840520 · focus · HN ↗

         There are several aspects to this:

         1. Better hardware optimized for matrix operations and other ML/NN specific workflows (NVIDIA hardware, Apple M4/M5 chips, Intel AI Boost, other NPU/TPU chips, etc.).

         2. Bigger RAM/VRAM/unified for loading models in memory -- with 3090/4090 you can run 22-24B models in VRAM; with 5090 you can run 30-32B models; with Apple hardware with 128GB unified RAM you can run 70-128B models.

         3. Better quantization (smaller, better value representations/lower errors/loss, optimized CUDA/etc. kernels) will lead to larger models being able to be run on the lower VRAM hardware at (with optimized kernels) decent tokens/s.

         4. Better optimizations in general taking advantage of the hardware (coupled with the hardware and driver advances) and the NN architectures (NOTE: I saw a huge performance boost a while back after upgrading my llama.cpp version).

         5. Better model architectures optimized/designed for running on lower-spec hardware, like Mixture-of-Experts which make it easier to run on a single GPU or distribute the weights more efficiently across multiple GPUs (i.e. you know that specific weights will be used together so you can keep them resident on a specific GPU and not have to swap the weights in/out of the various GPUs or transfer intermediate results of a single layer, only the result from that block/expert).

         5. Improved NVME or similar SSD technology which a) has a large amount of storage, and b) has access efficiency on par with RAM/VRAM, especially if/when coupled with efficient DMA access to the GPU/NPU.

Unofficial Hacker News client; not affiliated with Y Combinator.