Thread

Memory executives expect RAM shortage to continue through 2028

44 points · 66 comments · el_duderino

Loading the complete thread in the background. This saved snapshot is available now. Refresh

konaraddi · 1790944059 · focus · HN ↗
Discussed yesterday <a href="https://news.ycombinator.com/item?id=49920932">https://news.ycombinator.com/item?id=49920932
torutofu · 1790945970 · focus · HN ↗
yeah, at some point memory footprint becomes an architectural constraint again, not just a number you throw more hardware at.
1. jeffbee · 1790949585 · focus · HN ↗
  The memory capacity wall has been coming at us at high speed in plain sight for decades. We've been ignoring it.
  1. illini1 · 1790949815 · focus · HN ↗
    Doesn’t seem like it was coming at a high speed if it took decades to manifest.
  2. ajross · 1790952176 · focus · HN ↗
    Not sure I buy that? There's a hardware/process argument that DRAM scaling is hitting a wall (for largely the same reason that logic scaling is, even though the fabs are different).
    But DRAM has always been almost completely dominated by consumer device needs, and consumer requirements leveled off hard maybe a decade ago. Once you had a 4G system, things were mostly OK. Pervasive VM use and general bloat pushed that up by maybe a factor of two or three since, but that's it.
    AI is different. It wants that capacity for different reasons and puts it in different places. And almost none of us really saw that coming more than ~18 months ago or so. Certainly not to the extend that you'd conclude "we've been ignoring it".
  3. HighGoldstein · 1790960735 · focus · HN ↗
    How so? Memory bandwidth and capacity has been steadily increasing. On desktop for example, 10 years ago an i7-5960X was the absolute top of the line and it had a limit of 64GB, with most motherboards in practice being limited to 32-48GB, and this was an unthinkable amount of RAM at the time for desktop. Now the top end desktop chips support 192-256GB, and a year ago (before the RAM price spike) you could realistically buy that much if you wanted for a reasonable price.
    jeffbee · 1790963206 · focus · HN ↗
    The corresponding thread count has increased by much more than 4x.
    HighGoldstein · 1791026345 · focus · HN ↗
    Has it? The i7-5960X was 8C/16T, Ryzen 9 9950X (and variants) are 16C/32T. Even the Threadripper PRO 9000 WX is 96C/192T and supports a staggering 2TB of RAM.
2. testerius · 1790950175 · focus · HN ↗
  But still it is about GiB instead of KiB.
  1. samuelknight · 1790950401 · focus · HN ↗
    It's about 1TiB of HBM going into a GPU server eating about 3TiB of DDR that could have gone to consumer electronics.
  2. bunderbunder · 1790953252 · focus · HN ↗
    My health insurance company recently rolled out a major revamp of their website, and it now uses GiB as if they were KiB.
    I've switched from food blogs back to paper cookbooks because the ads eat up my tablet's memory and cause the browser to lock up and the page to repeatedly crash.
    At my knitting group last night one of the conversation topics was how everyone's computer feels really unstable these days. One person stopped volunteering at a local charity because, after a recent website rewrite, she has too much trouble with the page locking up when she's trying to sign up for hours.
    What I'm getting at is, if we do have to get used to memory being more expensive, that might become very disruptive because software developers will have to do a major about-face on how they've been writing software in the post-AI era.
    palmotea · 1790953960 · focus · HN ↗
    > What I'm getting at is, if we do have to get used to memory being more expensive, that might become very disruptive because software developers will have to do a major about-face on how they've been writing software in the post-AI era.
    I'm guessing you meant to say "pre-AI era."
    But don't kid yourself. What that means is right before they prompt "and don't make mistakes," they'll prompt "use less memory."
    joquarky · 1790959737 · focus · HN ↗
    You may joke but if it can be measured, it can be iterated upon.
    post-it · 1790954119 · focus · HN ↗
    On the other hand, memory optimization is so much easier with AI. Profiling an app, reworking a function to be twice as long but more efficient, A/B testing libraries across browsers to check which uses the least extra memory. All of these things are fairly easy but so tedious, which makes them perfect for AI.
3. vonneumannstan · 1790951484 · focus · HN ↗
  This is driven by AI and at a certain point you just need a minimum amount of memory to load all the floating points representing Neural Net weights.
4. amelius · 1790953470 · focus · HN ↗
  No more two-space copying garbage collectors.
Invictus0 · 1790948631 · focus · HN ↗
Can someone please explain to me how the recent llm cache breakthrough doesn't alleviate this memory shortage issue? <a href="https://intl.cloud.baidu.com/en/article/8937874" rel="nofollow">https://intl.cloud.baidu.com/en/article/8937874
1. gortok · 1790948715 · focus · HN ↗
  Not everyone is using baidu?
2. Grimeton · 1790949259 · focus · HN ↗
  I have this wild theory that this has not just todo with AI but with weapons production overall.
  1. palmotea · 1790954091 · focus · HN ↗
    > I have this wild theory that this has not just todo with AI but with weapons production overall.
    Why? Weapons are low-volume compared to commercial products, and they probably use trailing edge or weird bespoke components.
    I was digging around the other day, and at least the initial versions of the Tomahawk cruise missile had the computing power of an original IBM PC. That just shows how little computing power is needed for weapons capabilities.
    naishoya · 1790955704 · focus · HN ↗
    > how little compute power used to be needed ...
    ... for what were then 'advanced' weapons capability in high-cost units produced and expended in relatively small numbers with high yield payloads.
    The modern drone warfare version of expendable, high volume, high technology small units with two way communication via fibre-optic and RF jamming resistant multiple channels, and the ability to do on-device video object identification and terminal flight control when active electronic counter measures degrade operator control...
    That seems like exactly the kind of gear using commodity DDR4/5 sodimm and dimm modules and literally burning them up at then end of nearly every sortie. Even if they are using much smaller units (smartphone grade RAM) it's still cooking the limited output of the global memory fab capacity, and from the fields covered in fibre-optic blankets, I think it's safe to say there is a significant volume of electronics equipment being consumed.
    So yeah, AI might be ordering memory and putting it in datacenter, but eventually that material will be available for e-waste recovery (repurpose first, then recycle later). Both are competing for the same limited fab capacity.
    Grimeton · 1790970328 · focus · HN ↗
    Not just that. Everybody is rearming and the run for the first AI enabled fully autonomous robot is also going on.
    <a href="https://m.youtube.com/watch?v=TYaj0QnZx9s" rel="nofollow">https://m.youtube.com/watch?v=TYaj0QnZx9s
    naishoya · 1791001438 · focus · HN ↗
    Exactly. It might actually be a long term benefit to humanity that memory fab production is constrained at this juncture.
3. selbyk · 1790949356 · focus · HN ↗
  Why use less memory when you can just have more LLM?
  1. Invictus0 · 1790953968 · focus · HN ↗
    This is not a very useful comment
    post-it · 1790954278 · focus · HN ↗
    Yes it is, they're talking about induced demand. There are lots of resources where, when usage of the resource becomes more efficient, people end up using more rather than less.
4. linuxftw · 1790949642 · focus · HN ↗
  If training and inference is hardware constrained, and you can train and server better and bigger models on the same hardware with memory optimizations, that's exactly what I would expect companies to do.
5. s0ss · 1790949694 · focus · HN ↗
  Demand outstrips supply. Efficient software is great, but it doesn’t directly resolve insufficient supply of hardware.
6. moribvndvs · 1790950779 · focus · HN ↗
  See Jevons paradox. <a href="https://en.wikipedia.org/wiki/Jevons_paradox" rel="nofollow">https://en.wikipedia.org/wiki/Jevons_paradox
7. lostmsu · 1791196067 · focus · HN ↗
  This might regress intelligence.
maerF0x0 · 1790950139 · focus · HN ↗
More like they're praying it does while waiting for more options to vest
Bridged7756 · 1790950929 · focus · HN ↗
I got a 64gb 6000mt/s kit for 250 recently. Not a humble brag.
1. xtanx · 1790951394 · focus · HN ↗
  How?
  1. post-it · 1790953843 · focus · HN ↗
    $250 angle grinder
2. amelius · 1790953403 · focus · HN ↗
  Yeah, sure.
  1. iLoveOncall · 1790953722 · focus · HN ↗
    He didn't say it's a working kit.
cedws · 1790952300 · focus · HN ↗
What was the benefit of all this AI stuff again? Because all I see is middle and lower class being fucked over, repeatedly. Increasing prices of consumer electronics, job losses, suffering quality of goods and services, consolidation of power in favor of elites.
1. NewJazz · 1790952571 · focus · HN ↗
  Think about the app sitting on some developer's laptop, necer getting deployed anywhere else! Or the thousands getting uploaded to app stores to slurp and sell data!
vivzkestrel · 1790952513 · focus · HN ↗
or you can wait until one of those frontier labs IPO goes bust in a really bad way where it pumps 10x on day 1 and then everyone dumps every share out there and half a million people are holding bags. that ll surely wipe out all the AI hype
bunderbunder · 1790952773 · focus · HN ↗
Oracle's recent force majeure invocation shines a spotlight on what seems to be an endemic problem: datacenter builders have been signing commitments to bring a certain amount of capacity online by a certain date, without first conducting sufficient due diligence about whether they would be able to secure all the resources needed to meet those commitments.
This means they are now caught in a mad scramble to source all sorts of stuff, including RAM, potentially as a matter of life and death.
What's less clear to me, though, is whether datacenter builders will be able to make good on these purchase agreements. It seems these projects have every opportunity to fail, and the path to success keeps getting narrower. What are the odds that chip manufacturers ultimately get caught holding a bag of chips that nobody can buy in one hand, and a bag of IOUs from bankrupt customers in the other?
1. LetsGetTechnicl · 1790953187 · focus · HN ↗
  Yeah there was just a story about how there are hundreds of billions of dollars in "non-cancellable" debt obligations to build AI data centers. But they're building far beyond current demand, I feel like the collapse is inevitable.
  1. NooneAtAll3 · 1790954008 · focus · HN ↗
    sure, collapse is inevitable, alright
    how does one prepare for the moment it actually happens? what would it look like?
    what were the immediate consequences of dotcom for the normal folk?
    LetsGetTechnicl · 1790954176 · focus · HN ↗
    I'm sensing sarcasm but is it not true that the scale of obligations is unsustainable?
    As for preparation, I'm not sure. I have a 401k that I'm worried about. I'm too young to really remember the dotcom bubble, let alone have any stake in the stock market at the time.
    NooneAtAll3 · 1790955755 · focus · HN ↗
    It's half sarcasm, half my frustration in the way ai crash proponents behave in general
    if you proclaim that there's an inevitable doom - propose preparations *people you're talking to* can take
    when I hear about nuclear doom, I see preppers discussing bunkers and how rich people buy land in Argentina to evacuate to
    when I hear about stock market crash, I see people discussing some famous super-investor moving his money into coca-cola, because food would always have a demand
    even bloody "the end is nigh" apocalypse junkies continue with "go to church" (or smth similar) in the very next sentence
    what can YOU suggest ME as the next action? not for the government to do something, not for the whole market - how do you suggest ME to prepare? what do you suggest *I* will experience?
    at best, our approximation of ai bust is the dotcom - and that happened beyond current generation's memory. At best surviving context is "and the ruin left a lot of cheap fiber laid out all around the world", so... we'll get a lot of cheap ai then? good
    addaon · 1790956095 · focus · HN ↗
    Not every inevitability can be mitigated. We're each and every one going to die. The sun will die. The universe will wind down. Embrace it, don't pretend you can avoid it.
    mapontosevenths · 1790956151 · focus · HN ↗
    > at best, our approximation of ai bust is the dotcom - and that happened beyond current generation's memory.
    I'm so old now they think I'm dead. Honestly, it might be kind of a relief.
    NooneAtAll3 · 1790969535 · focus · HN ↗
    you're not dead - you just stopped being current
    just part of the riverbank :)
    bunderbunder · 1790956869 · focus · HN ↗
    Benjamin Graham’s all-purpose advice served at least some people well in the late 1990s: when stocks are expensive and bonds are cheap, buy bonds. When bonds are expensive and stocks are cheap, buy stocks.
    That doesn’t necessarily imply selling any investments you already have. Just what to favor for new investment.
    bigbuppo · 1790957020 · focus · HN ↗
    Except we don't get cheap AI this go round. It's just a bunch of e-waste that can't even be repurposed into server in the closet back at the office because each server needs about 100kW of power.
    sifar · 1790968501 · focus · HN ↗
    Nobody Knows. The market can remain irrational longer than you can remain solvent.
    Having cash in reserve to deploy in the aftermath of a crash is golden. If that happens you look great, if that doesn't happen for the next 5 years, you miss out on the interim gains.
    Keep investing, diversify, have some reserves. Try to be less anxious about things. Exercise, sleep, be grateful for things you have.
    danaris · 1791105447 · focus · HN ↗
    How about this:
    Don't try to retool your entire business, or personal life, around subscriptions to the big LLM companies.
    Continue to value human intelligence, skill, and work.
    Don't invest in companies that are retooling their entire business around subscriptions to the big LLM companies. Or, for that matter, in any company too tied up in the "AI boom". Yes, that might mean taking money out of big tech stocks for a little while.
    mapontosevenths · 1790956022 · focus · HN ↗
    Save money now.
    When the bubble bursts it will depress valuations across the board, including the valuations of the good companies that will survive to eventually own significant market share. Buy those at those depressed prices and wait a decade.
    The trick will be learning to tell the difference between the Google's and the pet.com's of the AI era. The trick during the dotcom situation was to look for companies that had actual gross profit and were reinvesting it, rather than companies that only had theoretical profit based on nonsense like market share and eyeballs.
    Take Amazon as an example. They were $107 a share in 1999. By late 2001, it had fallen to under $6. Now they are $250. It still took almost 10 years to recover.
    bigbuppo · 1790956865 · focus · HN ↗
    Worst part is that some of the companies that survived the dot com era are the new "...but on the internet" companies except now it's "...but with chat bot"
    LetsGetTechnicl · 1790960325 · focus · HN ↗
    Is there anyone in the AI sector that is making profit and re-investing it? I guess that really only applies to Microsoft, Apple, Google, etc. that other lines of business to prop up their AI divisions, vs. companies like OpenAI and Anthrophic. I wonder where NVIDIA will land...
    mapontosevenths · 1791000978 · focus · HN ↗
    Google is making money on it. However, that's coming at the expense of the very free web they built it from. So they have a different problem that may be unsolvable.
    kortilla · 1791089903 · focus · HN ↗
    Your Amazon example is misleading. That $250 price today really undersells the growth because that’s post split. Their dot com high was like $4 in today’s terms. Or multiply today’s price by 20 for the 1999 equivalent ($5000).
    It doesn’t negate your point about 10 years to recover but it’s important to highlight how substantial the growth was after it did.
    bunderbunder · 1790954428 · focus · HN ↗
    Losing one's retirement savings, mostly.
    I'm not going to give any financial advice. But I will say that in 2001 the people who, a few years earlier, thought to themselves, "Y'know, it looks like this much-anticipated rocketship to the moon has already been priced into these companies' stocks," didn't seem so stupid after all.
2. DrBazza · 1790954123 · focus · HN ↗
  There are signs that we are, and are not, in an AI bubble.
  Compared to the dotcom boom, firms are well positioned in terms of capital - they have revenue and profits.
  However, there's still the absurd speculative valuations, risk, and high expenditure (on said data centres). Also, Andrew Bailey from the Bank of England, personal opinion - <a href="https://www.bbc.co.uk/news/articles/cv8e30enrkxyo" rel="nofollow">https://www.bbc.co.uk/news/articles/cv8e30enrkxyo
ChrisArchitect · 1790953070 · focus · HN ↗
[dupe] <a href="https://news.ycombinator.com/item?id=49920932">https://news.ycombinator.com/item?id=49920932
Magi604 · 1790953099 · focus · HN ↗
Was at the computer store yesterday and the RAM I bought for $130 last year is now over $800.
1. tuvix · 1790953927 · focus · HN ↗
  Yeah I checked on DDR4 prices too recently and it made me pretty sad. Glad I upgraded to 64 GB in 2023 though. Another homelab dream deferred
RobotToaster · 1790953760 · focus · HN ↗
That's assuming CXMT or another Chinese company doesn't pull something out of the bag.
1. HighGoldstein · 1790960212 · focus · HN ↗
  Watch as memory manufacturers learn nothing from the car industry, and cry for tariffs when the Chinese inevitably annihilate them.
  1. nullocator · 1790961561 · focus · HN ↗
    Or possibly more likely get the Chinese manufacturers labeled as some kind of national security threat and forbid importing their products.
exabrial · 1790954125 · focus · HN ↗
/s/execpt/create/
mmmBacon · 1790954444 · focus · HN ↗
Remember when software was eating the world? The gravy train ran out on commodity hardware and now we are paying the price for almost 3 decades of underinvestment.
sharts · 1790955256 · focus · HN ↗
Executives expecting $$ as usual
Roark66 · 1790960777 · focus · HN ↗
The real problem is not that they are buying all the hardware, but that they are buying it and keeping it in bloody warehouses without bringing it online. Because they haven't got DC capacity for it.
Has anyone else here been trying to rent some cloud gpus for reasonable money? I have. I have signed up to a dozen "providers" that advertise everything from tesla cards to b200 only to find stuff is either out of capacity or double/triple the price.

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.