Thread

1. torutofu · 1790945970 · focus · HN ↗

   yeah, at some point memory footprint becomes an architectural constraint again, not just a number you throw more hardware at.

   1. jeffbee · 1790949585 · focus · HN ↗

      The memory capacity wall has been coming at us at high speed in plain sight for decades. We've been ignoring it.

      1. illini1 · 1790949815 · focus · HN ↗

         Doesn’t seem like it was coming at a high speed if it took decades to manifest.
      2. ajross · 1790952176 · focus · HN ↗

         Not sure I buy that? There's a hardware/process argument that DRAM scaling is hitting a wall (for largely the same reason that logic scaling is, even though the fabs are different).

         But DRAM has always been almost completely dominated by consumer device needs, and consumer requirements leveled off hard maybe a decade ago. Once you had a 4G system, things were mostly OK. Pervasive VM use and general bloat pushed that up by maybe a factor of two or three since, but that's it.

         AI is different. It wants that capacity for different reasons and puts it in different places. And almost none of us really saw that coming more than ~18 months ago or so. Certainly not to the extend that you'd conclude "we've been ignoring it".

      3. HighGoldstein · 1790960735 · focus · HN ↗

         How so? Memory bandwidth and capacity has been steadily increasing. On desktop for example, 10 years ago an i7-5960X was the absolute top of the line and it had a limit of 64GB, with most motherboards in practice being limited to 32-48GB, and this was an unthinkable amount of RAM at the time for desktop. Now the top end desktop chips support 192-256GB, and a year ago (before the RAM price spike) you could realistically buy that much if you wanted for a reasonable price.

         1. jeffbee · 1790963206 · focus · HN ↗

            The corresponding thread count has increased by much more than 4x.

            1. HighGoldstein · 1791026345 · focus · HN ↗

               Has it? The i7-5960X was 8C/16T, Ryzen 9 9950X (and variants) are 16C/32T. Even the Threadripper PRO 9000 WX is 96C/192T and supports a staggering 2TB of RAM.
   2. testerius · 1790950175 · focus · HN ↗

      But still it is about GiB instead of KiB.

      1. samuelknight · 1790950401 · focus · HN ↗

         It's about 1TiB of HBM going into a GPU server eating about 3TiB of DDR that could have gone to consumer electronics.
      2. bunderbunder · 1790953252 · focus · HN ↗

         My health insurance company recently rolled out a major revamp of their website, and it now uses GiB as if they were KiB.

         At my knitting group last night one of the conversation topics was how everyone's computer feels really unstable these days. One person stopped volunteering at a local charity because, after a recent website rewrite, she has too much trouble with the page locking up when she's trying to sign up for hours.

         Last weekend I got talking to another parent at my kid's gymnastics class about how her computer's really struggling since they rolled out some "modernization" rewrites of internal business applications, and it's making it hard to get her work done.

         What I'm getting at is, if we do have to get used to memory being more expensive, that might be very disruptive. Programmers will have to do a major about-face on how they've been writing software in the post-AI era.

         1. palmotea · 1790953960 · focus · HN ↗

            > What I'm getting at is, if we do have to get used to memory being more expensive, that might become very disruptive because software developers will have to do a major about-face on how they've been writing software in the post-AI era.

            I'm guessing you meant to say "pre-AI era."

            But don't kid yourself. What that means is right before they prompt "and don't make mistakes," they'll prompt "use less memory."

            1. joquarky · 1790959737 · focus · HN ↗

               You may joke but if it can be measured, it can be iterated upon.
         2. post-it · 1790954119 · focus · HN ↗

            On the other hand, memory optimization is so much easier with AI. Profiling an app, reworking a function to be twice as long but more efficient, A/B testing libraries across browsers to check which uses the least extra memory. All of these things are fairly easy but so tedious, which makes them perfect for AI.
   3. vonneumannstan · 1790951484 · focus · HN ↗

      This is driven by AI and at a certain point you just need a minimum amount of memory to load all the floating points representing Neural Net weights.
   4. amelius · 1790953470 · focus · HN ↗

      No more two-space copying garbage collectors.

Unofficial Hacker News client; not affiliated with Y Combinator.