Thread

1. lil-lugger · 1790019853 · focus · HN ↗

   Is there anyone serious who thinks that the future is local models anyway? All computers used to be the size of rooms like these data centers and then they got smaller and faster until the home computer came. Is that not a possibility down the line as we improve efficiency of the models and increase compute?

   1. latchkey · 1790020406 · focus · HN ↗

      Home computers are still nowhere close to as powerful as a $500k+ 10kW server full of 1.5TB of HBM GPU compute.

      1. Zetaphor · 1790021085 · focus · HN ↗

         You don't need a data center to get real work done. Not every task requires "PhD level intelligence"

         1. latchkey · 1790021330 · focus · HN ↗

            That's the usual response, along with "you can't compete with free". But, look at how much money the frontier models are printing, it is obvious that the bell curve of usefulness is still centered around them.

            Let's also not forget that even the open models are not getting smaller, they are getting larger. Of course, you can distill them down into something that will fit on smaller compute, but at the end of the day, the data centers of compute, still play a huge role.

            1. ryanbrunner · 1790021970 · focus · HN ↗

               An easy counterargument is that the frontier models are swallowing all of the attention and money because they currently don't have a constraint of demonstrating that their value exceeds their cost. Take that away and it might turn out that cheaper models that are "good enough" and can be profitable are the more attractive route.

               1. Zetaphor · 1790022228 · focus · HN ↗

                  Well said, they're competing on marketing rather than merit. Everyone defaults to the frontier instead of exploring more optimal options
               2. latchkey · 1790022377 · focus · HN ↗

                  > they currently don't have a constraint of demonstrating that their value exceeds their cost

                  There is no loyalty in AI. I can switch to another model at near zero cost. They absolutely do have to demonstrate value. The concept of "good enough" is a misnomer because we're talking about putting these products into the hands of people who need to generate value from them.

                  1. ryanbrunner · 1790126725 · focus · HN ↗

                     They have to demonstrate value absolutely, but you're forgetting about the denominator of cost, and to a large degree so is the industry right now.

                     To take this argument to an absurd level, if Fable generating a feature cost $100,000 and Sonnet generating the same feature cost $1, it wouldn't be enough that Fable is marginally, significantly, or even dramatically better, because the cost difference is so outsized. For a variety of reasons, the true cost of tokens for frontier models is downplayed at multiple points (whether it's through AI companies running off funding, companies prioritizing AI adoption over AI costs, etc.)

                     If cost is not an issue at all, the best thing is always preferable regardless of cost. But in most cases, and AI is no exception, eventually cost will be an issue.

            2. Zetaphor · 1790022170 · focus · HN ↗

               Open models keep getting bigger, but smaller open models also keep getting smarter. What I can do with an 8b used to require a 32b.

               Also the decision makers who are signing off on things like ChatGPT Enterprise are at least 18 months behind the curve of what you can actually do with these things and how cheap they can be. They're still trying to figure out how to actually adopt the tech out of a sense of fomo, nevermind making nuanced decisions about hosting an open weights model. I see this firsthand in my own job.

               I'm talking about adding facts to a model by modifying engrams or trying to bolster guardrails with J-washing, meanwhile they're still trying to figure out how to best prompt Copilot.

               Give it a few years for everyone else to catch up, I'm barely able to catch my breath before there's some new development in the open source/weights space

               1. latchkey · 1790023015 · focus · HN ↗

                  > Open models keep getting bigger, but smaller open models also keep getting smarter.

                  Not only bigger, but smarter and more capable. From what I can tell, smaller are only getting smarter in very specific areas. There is a subtle difference there, that is extremely important.

                  > What I can do with an 8b used to require a 32b.

                  What exactly do you do with an 8b? I usually ask this question and either get no response or it is something that doesn't generate anything of value. So, please surprise me.

                  1. Zetaphor · 1790024218 · focus · HN ↗

                     Smaller models have improved in every regard, not just domain specific challenges. You can see this in the benchmarks or by just testing one yourself side by side with an older release.

                     I'm using it at work in multiple data classification and redaction pipelines. It's replaced tedious manual labor and opened that staff up to focus on the parts of the work that requires their human intuition, rather than spending time on tedium.

                     Presumably part of the reason you don't get a response is your hostile approach to asking.

                     1. latchkey · 1790024904 · focus · HN ↗

                        They have not improved in the one way that I use them today, which is coding. It is like talking to a halfwit, and I always end back up with codex.

                        Asking a direct question is not hostile. I'm glad to hear you've found a use for a smaller model that generates value. It gives me hope for the future, that said, I think we are still a long ways away from needing HPC in DC's.

                        1. Zetaphor · 1790027158 · focus · HN ↗

                           Different people have different requirements. I use my local model for coding regularly. It doesn't help the conversation that things like quant, config, harness, and even prompting are equally as important to your outcomes as the choice of model.

                           1. latchkey · 1790028679 · focus · HN ↗

                              I pay $200/month and I generate 10000000x more value of it than that. As you're saying, the actual barrier isn't just the model, it is the set up. As long as the walls are too high, $200 is going to dominate.

                              The internet was great before AOL, but the simplification is what brought the masses to it.

                              1. qlte · 1790039087 · focus · HN ↗

                                 You generate 2 billion dollars a month in value from a single Codex Pro plan?

                                 1. latchkey · 1790039299 · focus · HN ↗

                                    Cute. That said, if my business does as well as I think it will it'll be worth a lot more than that (not monthly).

Unofficial Hacker News client; not affiliated with Y Combinator.