This comes at a perfect time. The public doesn't need computers, they can just use their phones.
If the consumer had access to this RAM, they might all just run local or semi-local AI. It's important to outbid them so you can rent AI to them, and extract money from them in a million other ways while they use it.
Make RAM half the price it was before and you'd still have very few people trying to do local AI on it because it's just so damn slow for the task. Having 12 channels of DDR5 in an Epyc system with NVMe storage is still one of the slowest ways I can run LLMs locally.
No one is running cpu-only inference anymore. Deepseek flash / Qwen flash next et al only need ~12-24 gb of VRAM for the active experts, so anyone with a decent gpu + lots of RAM + server cpu can run them at okay speeds.
Or one can also run Strix Halo / DGX spark / Apple. They handle MoEs well and use normal (not HBM) ram.
some benchmarks from locallama:
Deepseek V4:
"~25t/s decode at full quant, no speculative decoding, 2x3090 (only one being used for this model) + 9684x w/ 12 channel ddr5 4800, latest llama.cpp"
"Getting about 300 tk/sec pp and 23 tk/sec decode on Mac M3 Ultra 256 GB. Using original full precision model weights."
pessimizer · · focus · HN ↗
If the consumer had access to this RAM, they might all just run local or semi-local AI. It's important to outbid them so you can rent AI to them, and extract money from them in a million other ways while they use it.
zamadatix · · focus · HN ↗
upboundspiral · · focus · HN ↗
No one is running cpu-only inference anymore. Deepseek flash / Qwen flash next et al only need ~12-24 gb of VRAM for the active experts, so anyone with a decent gpu + lots of RAM + server cpu can run them at okay speeds.
Or one can also run Strix Halo / DGX spark / Apple. They handle MoEs well and use normal (not HBM) ram.
some benchmarks from locallama:
Deepseek V4: "~25t/s decode at full quant, no speculative decoding, 2x3090 (only one being used for this model) + 9684x w/ 12 channel ddr5 4800, latest llama.cpp"
"Getting about 300 tk/sec pp and 23 tk/sec decode on Mac M3 Ultra 256 GB. Using original full precision model weights."
Qwen-Flash-Next: <a href="https://www.reddit.com/r/LocalLLaMA/comments/1wom3fe/qwen_38_flash_next_q4_k_m_130k_context_q8_cache/" rel="nofollow">https://www.reddit.com/r/LocalLLaMA/comments/1wom3fe/qwen_38...
<a href="https://www.reddit.com/r/LocalLLaMA/comments/1vcaztx/what_speeds_are_everyone_getting_with_deepseek_v4/" rel="nofollow">https://www.reddit.com/r/LocalLLaMA/comments/1vcaztx/what_sp...