Thread

1. sroussey · 1789769479 · focus · HN ↗

   Someone really needed a few hundred TB to waste on inference and went looking under the rugs…

   1. why_only_15 · 1789769980 · focus · HN ↗

      CPU DRAM can't really be used for inference efficiently -- inference mostly wants memory bandwidth, not memory capacity, and GPU DRAM has >10x more bandwidth. The fabs can switch between them but you can't switch after the fact.

      1. halJordan · 1789773542 · focus · HN ↗

         The majority of inference actually does happen in cpu.

         1. why_only_15 · 1789984229 · focus · HN ↗

            This is just wrong unless you're using some confusing definition. Notice that companies trying to do lots of inference aren't looking for CPUs, they're looking for GPUs/TPUs.

            1. halJordan · 1790560940 · focus · HN ↗

               Ah well, the opposite actually. Your definition, while traditional, is wrong. It was right some time ago. But it's wrong now. At least according to Intel and Nvidia in their disclosures. Intel was literally talking about their sales to "companies that do lots of inference" when they said it... so...

Unofficial Hacker News client; not affiliated with Y Combinator.