I Picked a threadripper for a box that needed a lot of IO and figured that with that many PCI lanes I couldn't go wrong. But I have to admit I've been more than pleasantly surprised by the performance of the CPU as well, it - easily - outperforms all of the XEON and I7 based boxes that I have. The only thing I wished I would have done different is to max it out with 256G DIMMs when they weren't the price of a car.
Zen 5 really is nice. I swapped my RAM from a Xeon W Sapphire Rapids machine into a Threadripper Pro 9000 series machine and I get almost double the memory read performance, plus it's a heck of a lot faster in single and multi core performance, and it runs cooler and quieter. Huge win all around. Aside from the price... (I went from Xeon W5-3435X to TR Pro 9985WX, eep.)
I went the opposite way and went back to Intel after many many years on AMD. I snagged a Xeon 654 Granite Rapids ($850) for a new workstation geared towards local inference. I don’t need a ton of CPU cores and all Granite Rapids have 8 memory channels where as you need very high end Threadrippers (9975wx for $4000) for an equivalent due to their CCD design.
As a bonus Granite Rapids is super power efficient and runs much cooler than my previous TR and Ryzens. Intel seems to be doing good things again! My only minor complaint is that P2P doesn’t work on my multiple GPU setup because every PCIe5x16 lane has its own dedicated root to the CPU, but all that bandwidth is useful for MoE models that are offloaded to RAM.
You could get a PCIe switch backplane, stick your GPUs on there and then have them communicate locally through the switch using p2p. For some GPUs you may have to use a hacked driver (google p2p 3090 for instance).
I wanted to do that, but didn't like any of the PCIe switches / backplanes I found (mostly due to price). Now, it might be worth buying one of these instead of trying to build a fancy PC. <a href="https://c-payne.com/products/pcie-gen4-switch-5x-x16-microchip-switchtec-pm40100?variant=44506997326091" rel="nofollow">https://c-payne.com/products/pcie-gen4-switch-5x-x16-microch...
C-Payne makes excellent stuff but it is very pricey and supply is more than a little spotty, I've used pretty much all of his stuff by now and I am a very satisfied customer if I can order what I need, which more often than not is not the case.
A good alternative is ADT, they make excellent boards, there is a 4 and a 5 slot PCIe expander with a 88096 on it that works extremely well. I have two of these connected to my rig with their own power supply and four GPUs in each (and another two in the main machine). It's not exactly a portable affair (to put it mildly). Note that these won't work in a standard PC case due to the slot spacing so you'll have to rig something for that yourself (I use 2020 + some custom 3D printed fixtures).
Yeah, I had to go high-end to get one with enough CCDs to make use of the memory channels. I'm curious to know what all-read bandwidth you see when you run intel mlc on your system. My Xeon W5 was doing ~180GB/s after a lot of tuning, which was very disappointing. The Threadripper does ~320GB/s. From what I've heard, Granite Rapids is supposed to have solved the bandwidth problems that ailed Sapphire Rapids, so your numbers ought to be similar.
jacquesm · · focus · HN ↗
Gracana · · focus · HN ↗
rubiquity · · focus · HN ↗
As a bonus Granite Rapids is super power efficient and runs much cooler than my previous TR and Ryzens. Intel seems to be doing good things again! My only minor complaint is that P2P doesn’t work on my multiple GPU setup because every PCIe5x16 lane has its own dedicated root to the CPU, but all that bandwidth is useful for MoE models that are offloaded to RAM.
jacquesm · · focus · HN ↗
Gracana · · focus · HN ↗
jacquesm · · focus · HN ↗
A good alternative is ADT, they make excellent boards, there is a 4 and a 5 slot PCIe expander with a 88096 on it that works extremely well. I have two of these connected to my rig with their own power supply and four GPUs in each (and another two in the main machine). It's not exactly a portable affair (to put it mildly). Note that these won't work in a standard PC case due to the slot spacing so you'll have to rig something for that yourself (I use 2020 + some custom 3D printed fixtures).
Gracana · · focus · HN ↗