Thread

1. dghlsakjg · 1785816328 · focus · HN ↗

   I know everyone wants to crap all over these setups that are impractical, but this is how progress happens.

   People will keep plugging away at this and figure out how to avoid wearing the hard drive, how to make it run faster, custom hardware buses etc.

   Keep going! I personally can't wait for the day when a 1t param model runs off a $200 SSD instead of a $50k rack of Nvidia chips.

   1. gpt5 · 1785824544 · focus · HN ↗

      Most people are already used to rely on the internet on basically everything. At best, they download a tiny chunk of entertainment from it when they go on a plane, and as soon as they land they immediately abandon that offline chunk.

      In addition, LLMs, small or large, are highly parallelizable. This means that running on the same machine/GPUs many requests in parallel is significantly more efficient, and the sum of tok/s will far outweight any single user use case.

      Those two combined means that unless LLMs reach the state of 'good enough' (TBD), I fully expect the economics and people's interest to align with 99%+ of LLM usage would be in centralized servers. (not dismissing the fact that there are use cases for local LLMs, and that the progress matters).

      1. anon373839 · 1785825976 · focus · HN ↗

         I suspect the economics favor centralized servers, if you only look at the aggregated cost to serve X number of users' tokens. But we could say the same thing about a lot of the computation that iPhones do locally. They could have been much thinner clients, but instead they now have more compute power than desktops had when iPhones launched.

         1. gpt5 · 1785827087 · focus · HN ↗

            iPhone are in practice more reliant on central servers than ever before. Except for some games, if you take a random person's iPhone it becomes almost useless without internet connection. Which is exactly my point, it's not about compute, it's about people's expecations and priorities. The main driver of the need for higher performance chips on iPhones is its camera and smooth 120fps UI animations, not local processing for apps for almost all people.

            However, the compute situation just skew it even further when it comes to LLMs.

            1. anon373839 · 1785827772 · focus · HN ↗

               Phones are communication devices. So I would always expect connectivity to be central to their use. But that's a different issue than the "inefficiency" of edge computation and whether powerful edge devices can sell (they obviously can).
         2. xyzsparetimexyz · 1785830146 · focus · HN ↗

            A lot of the compute that iPhones do either has high bandwidth requirements (photo editing etc) or low latency requirements (games etc). LLMs are an interesting case where unless you're feeding in images, it's just text
         3. zozbot234 · 1785831394 · focus · HN ↗

            > I suspect the economics favor centralized servers, if you only look at the aggregated cost to serve X number of users' tokens.

            The economics of real-time, low-latency inference of very large near-SOTA models will heavily favor a centralized setup. But if you can afford to wait for your answer - be it a day, a week, or even more at the extreme low end (or if you just stick to leaner models for your relatively quick replies) the economics start to shift in a very clear way. A slow-going local inference setup relying on cheap SSD offload does not need the high power input of a datacenter rack, and the cooling load is outright trivial - even when working on many requests in parallel, which (in a SSD offload context) is what maximizes throughput even for local inference. These are serious problems for centralized inference that will probably limit the scale at which it can be applied.

         4. dghlsakjg · 1785980032 · focus · HN ↗

            iPhone, at this very moment, ships with an llm that third party developers can use. People are already using local llms.
      2. submeta · 1785829895 · focus · HN ↗

         I don’t understand the downvotes. It’s perfectly reasonable to disagree, but the argument is laid out clearly. Why not engage with it instead?
      3. mihaaly · 1785834857 · focus · HN ↗

         This is a nice personal point of view. And you don't care much about privacy or being exposed to the conduct of others completely, do you?
      4. hobofan · 1785835762 · focus · HN ↗

         > Those two combined means that unless LLMs reach the state of 'good enough'

         Once you are able to not just go on-premise, but on-device, there a boatload of other advantages than efficiency (network independence, privacy, etc.) that make it attractive.

         Tiny language models are already plenty "good enough" for a lot of on-device use-cases. E.g. there are a lot of things, that can be reformulated as ranking problems, where you can apply pairwise ranking, which even small language models are good at.

      5. dudefeliciano · 1785843822 · focus · HN ↗

         > Most people are already used to rely on the internet on basically everything

         FAANG has trained them well. "Normal" people also use chatGPT & co. to ask mundane questions that qwen27b could easily answer (random trivia, writing emails, correcting translations etc), why do these simple requests need to be processed on a B300 on server on the other side of the planet? There is no reason.

         And this being hackernews, I would expect people here to go against the grain in this regard...maybe even promote local llms to "normies".

Unofficial Hacker News client; not affiliated with Y Combinator.