Thread

1. HarHarVeryFunny · 1789937583 · focus · HN ↗

   On a related note, I was reading yesterday that apparently the real bottleneck for Chinese production of AI accelerators is HBM production, not processors or ASML equipment.

   The lack of ASML EUV machines certainly hurts, and pushing DUV so hard results in abysmal yields of good chips, but you can compensate by running more wafers or making smaller chips, and the net result is that Huawei's Ascend production volume is limited by CXMT's HBM capacity not processor dies.

   The problem is that HBM manufacture requires many steps (die thinning, via drilling, plating, alignment) where the equipment used by everyone else (Samsung, SK Hynix, Micron) is also blocked by sanctions, so the Chinese are having to develop all of this themselves too, which they have, but yields are currently low, even when using shorter HBM stacks.

   1. andy_ppp · 1789938149 · focus · HN ↗

      Tokens per second is almost entirely memory bandwidth at inference time, training obviously needs more compute but you can add more chips for that.

      1. martinald · 1789952036 · focus · HN ↗

         Not quite, it's got quite a bit more complicated with agentic use cases.

         Prefill (input tokens) is heavily compute bound. And the ratio of input to output continues to rise, as typically in agentic sessions you have a few tokens output for a tool call and (many) thousands of input from the tool result.

         Then you have cached input tokens, which is a totally different issue, system RAM or NVMe bound.

         Obviously output tokens is VRAM memory bandwidth bound, but this is less and less of the bottleneck these days for overall agentic speed.

         1. com2kid · 1789960294 · focus · HN ↗

            I can easily use close to 100 million input tokens a day. A few million output tokens but at the end of the day maybe a thousand or so lines of code get written.
         2. andy_ppp · 1789986705 · focus · HN ↗

            This is why I was careful to specify tokens per second not time to first token which is the prefill step you’re talking about. Clearly to run these models well you need both but as I said adding more chips or compute units can give you more latency where as overall memory bandwidth (throughput) is limited by access to fast memory.

Unofficial Hacker News client; not affiliated with Y Combinator.