I don’t think picking a handful of SPEC benchmarks that approximate today’s most common agentic workloads (compiling code, interpreting Python) and then calling them “agentic benchmarks” is misleading at all.
That you need a whole lot of “ordinary” compute to benefit from the scaling properties of agents is the reason Nvidia is making this chip in the first place.
The four benchmarks selected are cppcheck, llvm, cpython, and gcc [1]. These are all essentially compiler benchmarks... and all of the compiler benchmarks in SPEC cpu2026! This makes the benchmark selection somewhat suspicious to me, since it's not particularly representative of a diverse set of workloads.
I also don't buy that it's a particularly representative set of tasks you might do with agents. Also included in the SPEC benchmarks are multimedia codecs, lossless data compression codecs, sqlite (i.e., database), all of which are going to be things you should easily throw into the sets of tasks an agentic workload might do. Cherry-picking just the compiler benchmarks instead of all of SPECint... again, it just raises a couple of eyebrows.
[1] To be honest, I'm kinda surprised that both gcc and llvm are in SPEC cpu2026.
The code in compilers is the closest to your typical app you can get in a benchmark like SPEC, eveerything else is actually far more specialized. Compiler code is full of small basic blocks, lots of branches, indirect memory access; it's actually harder to get good performance for such code, both for CPUs and compilers (that was part of the death of Itanium too).
I mildly disagree - depending on your definition of "typical app". Most applications have much greater use of multi-processing and concurrent cross-thread (or cross-process) communication. Compilers, aside from high-level parallelism across modules, tend to be quite single-threaded applications.
If you're solely interested in single-core performance, then I would agree that they are a good stress test, but I think for a processor that is being sold on it's parallelism, they are not a great benchmark.
In the history of the SPEC benchmarks, the compiler benchmarks, like gcc, have been the best predictor of CPU performance for the applications that cannot benefit from array operations, so they cannot use the vector or matrix instruction set extensions.
The reason is that for the other benchmarks the CPU vendors have always succeeded sooner or later, to tweak their compilers and compiling options, or even the hardware of the CPUs, in order to get improved benchmark results that nonetheless are not indicative of the improvements in other applications.
On the other hand, the compiling benchmarks, like with gcc, and now also with clang, are too diverse in CPU resource usage and no special feature of the CPU has a significantly greater weight than others, so special tricks to enhance the benchmark results have never been found.
When looking at the past SPECint results, the values of the gcc benchmark remain the most reliable relative performance estimator.
I doubt that this will change in the near future.
Moreover, the multi-threaded compiling benchmark is also very useful, because it matches exactly a real-world workload that is extremely frequently encountered. Due to the great clock frequency difference between running a benchmark on a single thread and running it on all available threads, the single-threaded results have a very poor correlation with the multi-threaded results.
twoodfin · · focus · HN ↗
That you need a whole lot of “ordinary” compute to benefit from the scaling properties of agents is the reason Nvidia is making this chip in the first place.
jcranmer · · focus · HN ↗
I also don't buy that it's a particularly representative set of tasks you might do with agents. Also included in the SPEC benchmarks are multimedia codecs, lossless data compression codecs, sqlite (i.e., database), all of which are going to be things you should easily throw into the sets of tasks an agentic workload might do. Cherry-picking just the compiler benchmarks instead of all of SPECint... again, it just raises a couple of eyebrows.
[1] To be honest, I'm kinda surprised that both gcc and llvm are in SPEC cpu2026.
fibonacci112358 · · focus · HN ↗
14113 · · focus · HN ↗
If you're solely interested in single-core performance, then I would agree that they are a good stress test, but I think for a processor that is being sold on it's parallelism, they are not a great benchmark.
adrian_b · · focus · HN ↗
The reason is that for the other benchmarks the CPU vendors have always succeeded sooner or later, to tweak their compilers and compiling options, or even the hardware of the CPUs, in order to get improved benchmark results that nonetheless are not indicative of the improvements in other applications.
On the other hand, the compiling benchmarks, like with gcc, and now also with clang, are too diverse in CPU resource usage and no special feature of the CPU has a significantly greater weight than others, so special tricks to enhance the benchmark results have never been found.
When looking at the past SPECint results, the values of the gcc benchmark remain the most reliable relative performance estimator.
I doubt that this will change in the near future.
Moreover, the multi-threaded compiling benchmark is also very useful, because it matches exactly a real-world workload that is extremely frequently encountered. Due to the great clock frequency difference between running a benchmark on a single thread and running it on all available threads, the single-threaded results have a very poor correlation with the multi-threaded results.