Thread

1. dghlsakjg · 1785816328 · focus · HN ↗

   I know everyone wants to crap all over these setups that are impractical, but this is how progress happens.

   People will keep plugging away at this and figure out how to avoid wearing the hard drive, how to make it run faster, custom hardware buses etc.

   Keep going! I personally can't wait for the day when a 1t param model runs off a $200 SSD instead of a $50k rack of Nvidia chips.

   1. arjie · 1785817257 · focus · HN ↗

      Haha 1T on $50k might be a bit hopeful, mate, even at FP8. But I too am hopeful.

      1. hedora · 1785817651 · focus · HN ↗

         AMD already demonstrated 1T on strix halo clusters. << $10K at original MSRP.

         1. arjie · 1785873952 · focus · HN ↗

            Problem is prefill on these, right? Initial prompt processing takes forever? I suppose you’re right. Cost is not a thing on its own. It’s a performance-cost frontier and one can do CPU inference in the worst case.
      2. apimade · 1785820168 · focus · HN ↗

         8800 GTX in 2006. Cutting-edge, an insanely powered consumer card for the time. Theoretically around 0.3456 TFLOPS.

         1080 GTX in 2016. Cutting-edge, an insanely powerful consumer card for the time. Theoretically around 8.87 to 8.9 TFLOPS.

         5090 RTX in 2026. Cutting-edge, an insanely powerful consumer card for today. Theoretically around 104.8 TFLOPS.

         In the same timeframe mobile processor CPU's went from 0.001 TFLOPS, to today's Apple's A19 Pro chip which delivers 2.074 TFLOPS.

         That's _without_ getting into ASIC's, or purpose-built hardware like Taalas's model on silicon HC1, or generic AI dies like what they're planning with HC2 or Cerebras, which will massively compress the timeline.

         1. cududa · 1785824681 · focus · HN ↗

            Just a note that I think the direction most people are paying attention to is memory bandwidth; thats the real bottleneck and “number go up” but also constraint people are designing around
         2. xbmcuser · 1785825410 · focus · HN ↗

            We will see such power and price now only when AI market crashes or China reaches node parity and goes after market share as currently the way they are buying out most of the latest node production the consumer prices will only be palatable to the very rich or we will need to be happy with older slower nodes
         3. formerly_proven · 1785825576 · focus · HN ↗

            Except the 499$ of a 1080 GTX inflation-adjusted only buys you a 5070 or 5070 Ti even by MSRP.
         4. zmmmmm · 1785825969 · focus · HN ↗

            Sadly while the FLOPS are increasing nicely, total graphics memory is stalled in consumer cards by comparison.
         5. jack_pp · 1785826217 · focus · HN ↗

            Isn't there such a thing as low hanging fruit?

            Aren't we already approaching theoretical physical limits? We're at 2nm

            1. kaashif · 1785828861 · focus · HN ↗

               (1) Yes.

               (2) Are you saying that you think we're at the limits of computing in general, or that specific technology?

               We know, for example, that a human brain level intelligence is possible to run on a human brain. We are nowhere near that. And actually that's not even a physical limit necessarily.

               But that is...not a low hanging fruit.

               1. fragmede · 1785829045 · focus · HN ↗

                  > We are nowhere near that.

                  Nowhere?

                  1. darkwater · 1785831720 · focus · HN ↗

                     Leaving aside the discussion on LLMs intelligence vs human intelligence, on a purely energy consumption level we are definitely and without any possible questioning nowhere near that indeed.
            2. cvak · 1785830224 · focus · HN ↗

               we are not at 2nm, we just call it that.
         6. root_axis · 1785826900 · focus · HN ↗

            Ok, now do memory capacity and bandwidth - the things that actually constraint local LLMs.

            1. apimade · 1785830928 · focus · HN ↗

               8800 GTX in 2006: 768 MB of GDDR3, with 86.4 GB/s of theoretical memory bandwidth.

               GTX 1080 in 2016: 8 GB of GDDR5X, with 320 GB/s.

               RTX 5090 in 2026: 32 GB of GDDR7, with 1.792 TB/s.

               This is fun, what's next?!

               PCI 8.0 is breaking 1TB/s, GDDR7 is 1TB/s.

               With just the _current_ timeline, things are looking like they'll compress once we get over this initial lump.

         7. flaburgan · 1785828300 · focus · HN ↗

            Yeah but here you describing the opposite phenomenon. You're saying that the hardware is going to become cheaper and more powerful with the years, to the point a current State of the Art model from today will run on a normal consumer hardware in ten years. What people are trying to do now is the opposite, optimize the software as much as possible so that it does not need the best hardware but the normal one we currently have. As if we were trying to make a current AAA game to run smoothly on the 1080 GTX of your example.

            1. dtj1123 · 1785835753 · focus · HN ↗

               No, both of these things can happen in parallel.

               The suggestion is that a 1T model could be made to run on cheap consumer hardware of the future.

         8. foxrider · 1785831977 · focus · HN ↗

            Speaking of ASICs - how likely is it that as models get better we'll see someone baking a whole model directly into the silicon? It's like having l0 cache.

            1. SJC_Hacker · 1785833482 · focus · HN ↗

               You could do it but there would be no point, The only advantage over would be power consumption. And it would be quite expensive.

               At the rate models are improving, it would be obsolete in six months.

               1. HPsquared · 1785834798 · focus · HN ↗

                  Power consumption and latency are very important on mobile

                  1. naasking · 1785835752 · focus · HN ↗

                     They're important everywhere of course, but especially on mobile. If AI researchers figure out how to offload knowledge and expertise from reasoning weights, then a core reasoning ASIC linked to the knowledge would totally rock.
               2. foxrider · 1785843013 · focus · HN ↗

                  Yes, right now it would be obsolete in six months, but I also must add that this never stopped crypto miners from making new ASICs. However, with how useful Kimi is right now - at some point if someone makes a dedicated hardware board with "good enough" model for daily tasks - that would be a very sought after commodity.
            2. kaelwd · 1785835060 · focus · HN ↗

               Only 8B currently but it&#x27;s been done: <a href="https:&#x2F;&#x2F;taalas.com&#x2F;products&#x2F;" rel="nofollow">https:&#x2F;&#x2F;taalas.com&#x2F;products&#x2F;

               1. apimade · 1785835955 · focus · HN ↗

                  The only _public_ example we know.

                  This is definitely being done with private models by HFT&#x2F;quant firms, data processing agencies&#x2F;orgs (large intelligence agencies, _every_ data analytics org, etc).

         9. Azantys · 1785839857 · focus · HN ↗

            Here you have shown yourself that progress slows down and doesnt speed up. 8.9&#x2F;0.35 = ~25x more performance in 10 years from 2006 to 2016. 104.8&#x2F;8.9 = ~12x more performance in 10 years from 2016 to 2026. Growth has dropped 50%.

            1. dghlsakjg · 1785850261 · focus · HN ↗

               That isn&#x27;t deceleration, you&#x27;ve just chosen a very selective way to compare. If you use time as a denominator, which is kind of intrinsic when talking about rates of acceleration, you get a very different result. If you graphed .3, 8.9, and 104.4 on the y axis, with years on the x axis, it would be pretty clear that there was in increase in the rate of progress.

               We went from adding 8 teraflops in a decade, to adding almost 100 the next decade. If we add &quot;only&quot; 400 more teraflops in the next decade the graph will make that initial growth look flat in comparison, even though your math would show that we are basically stalled out.

               It’s like claiming that a company that goes from making $1 to $1k to $100k to $1mm in a 4 year period has decelerating growth.

               1. ksec · 1785864224 · focus · HN ↗

                  &gt;It’s like claiming that a company that goes from making $1 to $1k to $100k to $1mm in a 4 year period has decelerating growth.

                  Because it is decelerating growth. There is a reason why we use YoY percentage in annual and financial reporting.

      3. sh3rl0ck · 1785830886 · focus · HN ↗

         We&#x27;d bought 4 x $11K Mac Studios at my college and via exo, we had Kimi K2.5 at 30 TPS.

         Not too wild an idea!

         1. arjie · 1785873873 · focus · HN ↗

            30 tok&#x2F;s generation? Nice! What was prefill by the way? Also, these are 256 GB RAM studios? Good timing on those!

Unofficial Hacker News client; not affiliated with Y Combinator.