Thread

1. metalspot · 1790094725 · focus · HN ↗

   I have done a fair amount of low level performance optimization with Opus 5 and its reasoning is still very poor. Like why is CRC so slow and going through loops until I ask it if is using hardware instructions and it tells me it is using its own hand coded implementation poor. Reasoning about l1/l2/l3 cache hit ratios and their implications basically throwing darts at the wall, in the wrong room. If you give it a benchmark feedback loop then it might get there eventually but still massive alpha for low level systems engineers who instinctively know how this stuff works and can now automate 99% of the grind.

   1. bee_rider · 1790098891 · focus · HN ↗

      > Reasoning about l1/l2/l3 cache hit ratios and their implications basically throwing darts at the wall, in the wrong room. If you give it a benchmark feedback loop then it might get there eventually but still massive alpha for low level systems engineers who instinctively know how this stuff works and can now automate 99% of the grind.

      I suspect a lot of the training set for this sort of thing is people online speculating about cache performance incorrectly.

      1. louthy · 1790103979 · focus · HN ↗

         Speaking as somebody who is a performance geek, my knowledge came from relentless experimenting over the years (starting in the 8 bit era). Beyond the basics, I haven’t seen much on high-performance engineering online. To learn, you need to do the hard yards and I think performance tweaking becomes almost instinctive rather than something driven by a hard set of rules.

         At a low enough level, every performance tweak becomes unique and bespoke.

         Of course, you could find people online talking about how to write high-performance code, but beyond a few basic techniques, their advice may not work for you — nobody can write a generalist article about performance engineering that will definitely solve the problem you have right now.

         Arguably, there are fewer patterns for an LLM to infer as highly optimised code tends to become more and more opaque in the search for a nanosecond here or there.

         1. cogman10 · 1790104976 · focus · HN ↗

            I don't disagree, but IMO, a lot of code doesn't get to the point where those very low level techniques drive performance. Like, yes, if you are doing some heavy floating point math then that's where you end up needing it. However, in a lot of code finding hot paths and often simply switching out a O(n^2) for an O(n log n) or faster.

            Getting and using tools to find hotpaths is generally the most important performance tweaking skill.

            1. louthy · 1790106114 · focus · HN ↗

               I mean, sure, but it really does depend on what you're doing. If you're working on a library with collection-types and you want to make each iteration as fast as possible, then roll up your sleeves. If you're writing a compiler and you want your language's source-code to finish compiling this week, roll up the sleeves. If you're working on a game-engine and you want to draw more than everyone else, roll up the sleeves...

               There are plenty of real-world reasons why you'd want to get knee deep in this stuff. I wasn't suggesting not using tools (I've literally spent the day buried in JetBrains' memory and tracing tools!), but those tools can only tell you what is happening now, not what to do to improve it.

               Profiling is, of course, essential. But performance tweaking can be quite a laborious process: if you're judging things by big-O notation, then that's a different level above the real low-level tweaking (imho of course). Picking the correct data-structures is all in the 101 of performance engineering. That's in the literature. But it's all too basic and simplistic. Most performance minded engineers wouldn't need a profiling tool to know which data-structure to use.

               At the smallest level there's a lot of mental theory building and experimentation as you try out different approaches, which is where the instinct and intuition starts to build. I never see any of that in discussions about performance engineering.

               1. cogman10 · 1790107777 · focus · HN ↗

                  > but those tools can only tell you what is happening now, not what to do to improve it.

                  They tell you what's happening now, but they also tell you if what you've done has had a positive impact.

                  > If you're judging things by big-O notation, then that's a different level above the real low-level tweaking (imho of course).

                  I completely agree. My point isn't that Big-Oh is low level, but rather that Big-Oh is often enough for most programming problems. Even in some of your examples like a compiler, game engine, or collection library, the big oh matters and if it's wrong, that can be a lot more important than shaving 0.1% on writing a function in a low level fashion. Big-Oh is gotten wrong a surprising amount of time even though it's 101 level stuff.

                  > At the smallest level there's a lot of theory building and experimentation as you try out different approaches, which is where the instinct and intuition starts to build. I never see any of that in discussions about performance engineering.

                  Oh because people get these things wrong all the time. That's why performance engineering stresses that you test, test, test and know what your testing and know why your testing could be wrong or corrupted. You should not trust your intuition because things change and it isn't always correct.

                  A good example of how easy it is to get measuring wrong. Imagine you start tweaking a function and you measure that your application became 1% faster. Was it the work you did on that function? Surprisingly, not always (at least not directly). Sometimes, it's the case that when you work on a function you re-align other functions as the machine code has to go it memory. It's possible that an undiscovered misaligned while loops was actually causing a large portion of your performance spill and by tweaking the function here, you aligned the while loop (or maybe a few of them). And, importantly, a new change somewhere else might re-unalign that same while loop.

                  You walk away thinking you've learn some low level lesson when in actuality your bit twiddling simply accidentally fixed something somewhere else.

                  This is why measuring is so important but also good measuring is even more important.

                  1. VorpalWay · 1790112293 · focus · HN ↗

                     > I completely agree. My point isn't that Big-Oh is low level, but rather that Big-Oh is often enough for most programming problems.

                     For what I work on, the hidden constant is often more important than big O. For example, a hash map has better complexity than just searching through a vector. But if the vector is small enough it will best the hash nap for actual time. Just plain searching until you find the element will even beat binary search on a sorted vector for small enough vectors. The reasons are complex, to do with cache, prefetch, branch prediction and also just how many instructions your tight inner loop has. (And the specific reasons will vary between desktop class CPUs and microcontrollers. But both exhibit this pattern.)

                     You could argue that at that point why bother optimising at all (there aren't a lot of elements in the collection after all). But there are two distinct cases I have come across over the years where it still matters (and for what I work with, they represent the common cases):

                     * You need to look up in a small collection a lot (either lots of lookups into a few small collections or a few lookups each into lots of different small collections, I have seen both cases).

                     * Hard realtime code where predictable latency matters. Hashmap has a bad worst case, binary trees and binary searching has badly predictable memory access patterns. And in this case the collections are usually small anyway (there are only so many actuators and sensors your equipment has, and/or the embedded microcontroller doesn't have a lot of memory anyway).

                     1. cogman10 · 1790171224 · focus · HN ↗

                        > But if the vector is small enough it will best the hash nap for actual time.

                        This will all depend on the size and type of object stored in a vector.

                        If you have a relatively small and flat struct that you are storing, then sure that will likely win. But if you are working with a collection of pointers, then the hash map will (almost) always win.

                        > Hard realtime code where predictable latency matters. Hashmap has a bad worst case

                        The hash map worst case is a linear search. It will be a lookup + the search. This also depends on the implementation. You could, for example, use a robinhood hash map which trades insertion times for lookup times.

                        A bad hashmap implementation will store collisions in linked lists. A better one will try and put them in a b-tree. An even better one will use a vector for collisions. And the most cache friendly version stores everything in the table and does probing on collisions.

                        There are specific cases where vectors are better, but those are the exception and not the rule in my experience.

                        1. VorpalWay · 1790245672 · focus · HN ↗

                           It all depends. I was thinking mainly of small keys that are at most a machine word. But yes I conceded that for large keys (extremely unusual for what I do, thus I didn't think of that case before) a hashmap wins.

                           Also, while on a desktop class CPU any pointer chasing you do tends to dominate, that is not the case on small and medium microcontrollers (which I work with a lot). They have very short pipelines and their memory is all internal SRAM most of the time. Here classic cycle counting is still king, so you need to ask yourself if it is quicker to compare the key than to calculate the hash and compare hashes.

                           Code size in general matters a lot on embedded. Even a simple hashmap will always have more code than a simple vector. And a state of the art implementation with probing and tombstones (such as swisstable in C++ and hashbrown in Rust) will be way too large to be usable.

Unofficial Hacker News client; not affiliated with Y Combinator.