Thread

1. _ache_ · 1789708917 · focus · HN ↗

   From my own test. It's not faster than the unsloth model.

   Disclarer: I'm unsing Vulkan on an AMD GC.

   1. DiabloD3 · 1789719037 · focus · HN ↗

      Surprised its not meaningfully slower.

      Vulkan and ROCm paths are missing a few optimized versions of the quants they're using.

      1. GrayShade · 1789748032 · focus · HN ↗

         I see around 1 tk/s after a while, so perhaps it is.
   2. Systemerror7A69 · 1789720479 · focus · HN ↗

      AMD 7900 XTX with Vulkan here as well, wasn't faster on my test either. Might be much different on Nvidia though.

      I assume the limit for me is memory bandwith, as the 7900 XTX has the same bandwith as the 3090 from what I can gather and I already reached ~60 t/s with Unsloth. Those would fit with the numbers Byteshape has for their cards.

      4090 and 5090 have much higher bandwith apparently, so on those cards you can probably get much more out of the kinds of performance improvements they are doing.

      1. noir_lord · 1789724361 · focus · HN ↗

         4090 isn't that much higher than the XTX (I also have the XTX), it's 1008GB/s (4090) vs 960GB/s for the XTX's.

         The 5090 destroys both at 1792GB/s.

         It's not really one thing with the nvidia cards best I can tell it's that they compounded incremental gains from software drivers, card kernels and optimization from been the primary choice (plus first mover advantage).

         I didn't buy the XTX for AI purely gaming but it's a capable enough local card for running Qwen et al.

         1. Figs · 1789733175 · focus · HN ↗

            4090 vs 5090 performance difference is largely GDDR6 vs GDDR7, I think

            1. wtallis · 1789748430 · focus · HN ↗

               It's equal parts memory clock and bus width: RTX 4090 is 384-bit wide at 21 Gb/s and RTX 5090 is 512-bit wide at 28 Gb/s.
      2. MrDrMcCoy · 1789869415 · focus · HN ↗

         Would you consider sharing your config? I can't get anywhere near that with my Strix Halo, nor my AI Pro 9700 XT.
   3. Schlagbohrer · 1789725455 · focus · HN ↗

      I feel weird that I like your typos, because clearly AI did not write your post. Typos have become downright charming and nostalgic for me.

      1. _ache_ · 1789750184 · focus · HN ↗

         Ahahah :'D

         Yes... I'm not english native. And definitively not an AI.

      2. _ache_ · 1789750838 · focus · HN ↗

         Ahahah, thank you.

         Yes, I'm not a native English speaker, and I'm definitely not an AI. ;)

         I can't say the same thing because I just don't notice typos (mine or others).j I just assume my English is bad.

         Isn't the definition of "being human" is "not to be perfect"? In French, it kinda is. We say "He/she humain after all" to mean that someone made a mistake.

Unofficial Hacker News client; not affiliated with Y Combinator.