I don’t think picking a handful of SPEC benchmarks that approximate today’s most common agentic workloads (compiling code, interpreting Python) and then calling them “agentic benchmarks” is misleading at all.
That you need a whole lot of “ordinary” compute to benefit from the scaling properties of agents is the reason Nvidia is making this chip in the first place.
The four benchmarks selected are cppcheck, llvm, cpython, and gcc [1]. These are all essentially compiler benchmarks... and all of the compiler benchmarks in SPEC cpu2026! This makes the benchmark selection somewhat suspicious to me, since it's not particularly representative of a diverse set of workloads.
I also don't buy that it's a particularly representative set of tasks you might do with agents. Also included in the SPEC benchmarks are multimedia codecs, lossless data compression codecs, sqlite (i.e., database), all of which are going to be things you should easily throw into the sets of tasks an agentic workload might do. Cherry-picking just the compiler benchmarks instead of all of SPECint... again, it just raises a couple of eyebrows.
[1] To be honest, I'm kinda surprised that both gcc and llvm are in SPEC cpu2026.
The code in compilers is the closest to your typical app you can get in a benchmark like SPEC, eveerything else is actually far more specialized. Compiler code is full of small basic blocks, lots of branches, indirect memory access; it's actually harder to get good performance for such code, both for CPUs and compilers (that was part of the death of Itanium too).
That's true of most of the applications in SPECint (SPECfp is a different matter); there's nothing special about compilers there.
Where compiler code is going to get really unusual, I suspect, is that compilers tend to be a little mono-focused on relatively few data structures. I know I was able to get measurable (single-digit percent!) performance differences in LLVM making very small tweaks to layout in llvm::Value. By contrast, when I was working on Thunderbird, the only similarly small change I could think to make that kind of difference would be to "oops, all string functions are now a cross-DLL call" (and even then, only because string handling is so dominant in that kind of application). Another kind of difference is that the compiler-based benchmarks are going to be quite light in virtual or indirect function calls (there's more of an emphasis on switch-based dispatching than vtable-based dispatching in most compiler implementations), which is going to make it a poorer proxy for some kinds of applications.
Seconding this - having worked on LLVM and Firefox, the performance tuning of each application was very different. Even measuring the performance of an application like Firefox (in a meaningful way) is non-trivial, wheras compilers are much more approachable with traditional profilers (either tracing or sampling).
Death of Itanium was that it was a) VLIW and b) Intel was too arrogant. So it went to the same destination as later Larrabee and ATI/AMD attempts at VLIW GPUs.
That is, nowhere.
Also you are wrong and anyone sizing up an arch to put their loads onto must first try that load on it and not rely on "bah, compilers compile on it".
VLIW works for DSP applications, it's not an instant dead end. It's a good fit in cases where code path and memory accesses are predictable, like shader code.
Essentially every cell phone out there has a VLIW DSP like Qualcomm's Hexagon cores (though AFAIK Qualcomm is the only one who lets you run your on programs on their DSP).
In a hypothetical world where AMD wasn't allowed to do AMD64, and Intel stayed committed to Itanium: Itanium would still have sucked, and both PowerPC and SPARC would have out-sold Itanium by an even wider margin than they did in this reality. Itanium could only have succeeded if AMD64 wasn't possible and literally all of the competing 64-bit architectures were killed off by their owners so they could jump on the Itanium bandwagon. Itanium managed to kill off PA-RISC and Alpha and (high-end) MIPS roadmaps, but it still had competitors that were not just viable but actually more successful.
Um...while not "OG Windows" I guess, PowerPC was an officially supported and shipping Windows NT 3.51 & 4 target[1]. It was released a couple of years before XP 64-bit for itanic. SPARC was a planned NT port, and while it never happened, it could have[2][3].
If you ignore VLIW, you just have a very unexciting RISC ISA, but because of the VLIW, you get extra scheduling info that most RISC designs don't provide which might be advantageous. The real question is actually about the code density of 41-bit instructions and if it can be offset by the 128-bit package (and perhaps something like allowing new 24-bit compressed instructions).
Poulson already somewhat proved part of this as it added back a traditional frontend and even added some OoO capabilities and 4-way SMT. It wasn't earth-shattering, but it wasn't absolute garbage either.
The contemporaneous IBM POWER ISA was implemented in superscalar CPU cores with out-of-order execution and with SMT and it would provide superior performance in any equivalent fabrication technology.
The Itanium ISA actually had a few nice features, but it also had other bad features that outweighed the good features. Besides the static instruction scheduling in bundles, there was also the handicap of using SPARC style register windows, which slowed-down context switches.
The second version of HP PA-RISC, which was too quickly replaced by Itanium, would have had good chances of providing superior performance in comparison with Itanium, had it not been abandoned without a fight.
I mildly disagree - depending on your definition of "typical app". Most applications have much greater use of multi-processing and concurrent cross-thread (or cross-process) communication. Compilers, aside from high-level parallelism across modules, tend to be quite single-threaded applications.
If you're solely interested in single-core performance, then I would agree that they are a good stress test, but I think for a processor that is being sold on it's parallelism, they are not a great benchmark.
In the history of the SPEC benchmarks, the compiler benchmarks, like gcc, have been the best predictor of CPU performance for the applications that cannot benefit from array operations, so they cannot use the vector or matrix instruction set extensions.
The reason is that for the other benchmarks the CPU vendors have always succeeded sooner or later, to tweak their compilers and compiling options, or even the hardware of the CPUs, in order to get improved benchmark results that nonetheless are not indicative of the improvements in other applications.
On the other hand, the compiling benchmarks, like with gcc, and now also with clang, are too diverse in CPU resource usage and no special feature of the CPU has a significantly greater weight than others, so special tricks to enhance the benchmark results have never been found.
When looking at the past SPECint results, the values of the gcc benchmark remain the most reliable relative performance estimator.
I doubt that this will change in the near future.
Moreover, the multi-threaded compiling benchmark is also very useful, because it matches exactly a real-world workload that is extremely frequently encountered. Due to the great clock frequency difference between running a benchmark on a single thread and running it on all available threads, the single-threaded results have a very poor correlation with the multi-threaded results.
I think the choices of these workloads are deliberate, considering this is a large core count CPU linked to a FP-monster GPU with a high speed, low latency datalink.
The former implies per core memory bandwidth is probably not great, meaning SQLite wont perform as well, the latter meaning FP workloads are better done on the GPU, so video encoding wont be a high point. The idea is to run branchy integer workloads that fit into RAM imo, which is what these benchmarks measure.
twoodfin · · focus · HN ↗
That you need a whole lot of “ordinary” compute to benefit from the scaling properties of agents is the reason Nvidia is making this chip in the first place.
jcranmer · · focus · HN ↗
I also don't buy that it's a particularly representative set of tasks you might do with agents. Also included in the SPEC benchmarks are multimedia codecs, lossless data compression codecs, sqlite (i.e., database), all of which are going to be things you should easily throw into the sets of tasks an agentic workload might do. Cherry-picking just the compiler benchmarks instead of all of SPECint... again, it just raises a couple of eyebrows.
[1] To be honest, I'm kinda surprised that both gcc and llvm are in SPEC cpu2026.
fibonacci112358 · · focus · HN ↗
jcranmer · · focus · HN ↗
Where compiler code is going to get really unusual, I suspect, is that compilers tend to be a little mono-focused on relatively few data structures. I know I was able to get measurable (single-digit percent!) performance differences in LLVM making very small tweaks to layout in llvm::Value. By contrast, when I was working on Thunderbird, the only similarly small change I could think to make that kind of difference would be to "oops, all string functions are now a cross-DLL call" (and even then, only because string handling is so dominant in that kind of application). Another kind of difference is that the compiler-based benchmarks are going to be quite light in virtual or indirect function calls (there's more of an emphasis on switch-based dispatching than vtable-based dispatching in most compiler implementations), which is going to make it a poorer proxy for some kinds of applications.
14113 · · focus · HN ↗
lstodd · · focus · HN ↗
That is, nowhere.
Also you are wrong and anyone sizing up an arch to put their loads onto must first try that load on it and not rely on "bah, compilers compile on it".
speed_spread · · focus · HN ↗
Symmetry · · focus · HN ↗
pjmlp · · focus · HN ↗
wtallis · · focus · HN ↗
pjmlp · · focus · HN ↗
Remember, the very first Windows XP 64 bit release was on Itanium.
my123 · · focus · HN ↗
kjs3 · · focus · HN ↗
[1] <a href="https://archive.org/details/NT351PMZPPC" rel="nofollow">https://archive.org/details/NT351PMZPPC
[2] <a href="https://www.techmonitor.ai/technology/undercurrent_bubbling_at_sun_to_get_windows_nt_implemented_for_the_sparc" rel="nofollow">https://www.techmonitor.ai/technology/undercurrent_bubbling_...
[3] Legend has it that the SPARC port existed, done by Intergraph, but for Reasons was never a product.
hajile · · focus · HN ↗
I'm not completely convinced of this.
If you ignore VLIW, you just have a very unexciting RISC ISA, but because of the VLIW, you get extra scheduling info that most RISC designs don't provide which might be advantageous. The real question is actually about the code density of 41-bit instructions and if it can be offset by the 128-bit package (and perhaps something like allowing new 24-bit compressed instructions).
Poulson already somewhat proved part of this as it added back a traditional frontend and even added some OoO capabilities and 4-way SMT. It wasn't earth-shattering, but it wasn't absolute garbage either.
adrian_b · · focus · HN ↗
The Itanium ISA actually had a few nice features, but it also had other bad features that outweighed the good features. Besides the static instruction scheduling in bundles, there was also the handicap of using SPARC style register windows, which slowed-down context switches.
The second version of HP PA-RISC, which was too quickly replaced by Itanium, would have had good chances of providing superior performance in comparison with Itanium, had it not been abandoned without a fight.
14113 · · focus · HN ↗
If you're solely interested in single-core performance, then I would agree that they are a good stress test, but I think for a processor that is being sold on it's parallelism, they are not a great benchmark.
adrian_b · · focus · HN ↗
The reason is that for the other benchmarks the CPU vendors have always succeeded sooner or later, to tweak their compilers and compiling options, or even the hardware of the CPUs, in order to get improved benchmark results that nonetheless are not indicative of the improvements in other applications.
On the other hand, the compiling benchmarks, like with gcc, and now also with clang, are too diverse in CPU resource usage and no special feature of the CPU has a significantly greater weight than others, so special tricks to enhance the benchmark results have never been found.
When looking at the past SPECint results, the values of the gcc benchmark remain the most reliable relative performance estimator.
I doubt that this will change in the near future.
Moreover, the multi-threaded compiling benchmark is also very useful, because it matches exactly a real-world workload that is extremely frequently encountered. Due to the great clock frequency difference between running a benchmark on a single thread and running it on all available threads, the single-threaded results have a very poor correlation with the multi-threaded results.
torginus · · focus · HN ↗
The former implies per core memory bandwidth is probably not great, meaning SQLite wont perform as well, the latter meaning FP workloads are better done on the GPU, so video encoding wont be a high point. The idea is to run branchy integer workloads that fit into RAM imo, which is what these benchmarks measure.