Thread

Clef: Open-weight decision models, and new RL fine-tuning platform

637 points · 217 comments · jasondavies

manlymuppet · 1790875177 · focus · HN ↗
Am I hearing this right, that they made a decision model based on Typesafe's new paradigm, and actually made a model better than Jev based on Typesafe's own ranking?
And it's only been a few weeks.
1. TeMPOraL · 1790876066 · focus · HN ↗
  It's not a "new paradigm", it's a low-hanging fruit that's been lying around for years; Typesafe were the first to bother to stop and pick it up, and market the shit out of it. But it was still a low-hanging fruit.
  There are many, many of those left around, because AI frontier is moving forward so fast, everyone is racing ahead. Which is why I laugh when people say AI is not transformative and LLMs are a dead end (and my favorite, "what are we going to do with all those GPUs when the bubble pops?"). Even if SOTA LLMs hit a hard capability limit tomorrow and never advanced again, there's a good decade of growth and advancement to be extracted just from all the low-hanging fruits that were left unpicked along the way.
  1. seizethecheese · 1790876112 · focus · HN ↗
    Name a few of these low hanging fruit left around.
    TeMPOraL · 1790876903 · focus · HN ↗
    Jev is one.
    Diffusion transformers are not "easy" but underfunded.
    Random one in terms of applications: getting GPT-4-level[0] LLMs to operate at hundreds of tokens per second on edge hardware - opens up so many possibilities I'm probably unable to imagine half of them.
    E.g. Imagine spellcheck/predictive text (or code autocomplete) where the model is able to process a whole paragraph + surrounding application/system context in between keystrokes. Or an OS being able to reliably guess what you're doing in real-time, in between your UI interactions, and offer actually helpful contextual reactions.
    Or imagine finally funding some decent studies into exploring the models as computational artifacts - studying their latent spaces, how they form and how they model reality internally.
    Or imagine automated sliding doors that don't suck.
    --
    [0] - Or anything substantially better than BERT-level models used in Jev or that demo from the company doing inference ASICs, that has a chatbot online that does 14 kilotokens per second.
    aeve890 · 1790877840 · focus · HN ↗
    >Random one in terms of applications: getting GPT-4-level[0] LLMs to operate at hundreds of tokens per second on edge hardware
    That's low hanging for you?
    blurbleblurble · 1790878356 · focus · HN ↗
    It's likely quite close. There are so many papers proving concepts that would bring this, they just haven't been combined in production.
    msdz · 1790878366 · focus · HN ↗
    Maybe they meant in the sense of “untapped potential”, because so far a lot of the focus has been on increasing model capabilities, not necessarily performance/power budget.
    TeMPOraL · 1790880844 · focus · HN ↗
    Yes. Point is, it's untapped only because everyone is running in the race (even if out of curiosity), and there's just not enough people with means to tap into these side threads. For the past few years, there's been many interesting papers that circulated the industry, got recognized as worthwhile pursuits, and then dropped because running behind the Big Vendors had massively better ROI.
    ekabod · 1790878678 · focus · HN ↗
    That's a high hanging fruit, not low.
    guyomes · 1790879773 · focus · HN ↗
    If we throw in hardware dedicated to a specific LLM, it seems to be a rather low hanging fruit. Especially considering that this is already happening for vision models [1].
    [1]: "FPGA-based CNN Acceleration using Pattern-Aware Pruning" <a href="https://inria.hal.science/hal-04689673/document" rel="nofollow">https://inria.hal.science/hal-04689673/document
    mdp2021 · 1790886835 · focus · HN ↗
    > hardware dedicated to a specific LLM
    That wording screams "Taalas". Which, importantly, is not the only player trying to abate the distance between data and arithmetics...
    TeMPOraL · 1790880673 · focus · HN ↗
    Yes. It's well within realm of possibility, but so far wasn't pursued because the Big Vendors went all-in into capability growth (rightfully testing "the bitter lesson" to its limits) and got themselves stuck in an arms race, while everyone else is barely keeping up and/or starstruck with fascination, exploring what these models can do.
    This got everyone racing forward and right now there is not enough human attention left in the world to productionize this, or any of the other "side threads". When the race slows down, people will catch up, branch out, and loop back.
    blurbleblurble · 1790881018 · focus · HN ↗
    Just like renewable energy and so many other things. Hyperconcentration of capital is really tragic. I hope things turn around.
    TeMPOraL · 1790881569 · focus · HN ↗
    They will. That's the fallacy of the "S-curve" everyone likes to commit these days actually giving a positive outlook.
    Assuming it won't get to full RSI, the current approach will burn out - most likely economically. The race slows down, people branch out, look back, start picking up the "untapped potential"/low-hanging fruits, and you have new S-curves launching in place of the one that just tapered off (hence a fallacy - a stack of S-curves adds up to continuing exponential growth).
    In other words: it comes and goes. Hyperconcentrated capital will eventually deconcentrate.
    mdp2021 · 1790884152 · focus · HN ↗
    > That's low hanging for you
    An important part of the industry is studying that: it is built-up effort. Sooner or later, the fruits will be harvested. The targeted preparation has been there for years now.
    Twirrim · 1790890577 · focus · HN ↗
    We already have examples of LLMs running 16k+ tokens a second using custom ASICs.
    It's down at the moment (Not sure if it'll return?) but Chat Jimmy[0] produced by Taalas[1] was powered by an ASIC running Llama 3.1 8B, and hitting 17,000 tokens/sec. It was amazing to use, you'd no sooner have hit enter than you had a full response back. I actually found its speed to be a problem for interactive stuff, as every answer got several paragraphs I'd then wade through, vs a model populating text closer to my reading speed.
    I appreciate, there are differences between an 8 billion parameter model and something GPT-4-ish, but we're currently in the middle of a race between a half dozen or so companies to produce the next best frontier model, which requires their infrastructure to be dynamic.
    We really don't always need newer better faster stronger models, there's quite a lot of room for "good enough" where getting 17kt/s at significantly lower power would be amazing.
    [0] <a href="https://chatjimmy.ai/" rel="nofollow">https://chatjimmy.ai/ [1] <a href="https://taalas.com/" rel="nofollow">https://taalas.com/
    AshamedBadger56 · 1790891575 · focus · HN ↗
    >I actually found its speed to be a problem for interactive stuff, as every answer got several paragraphs I'd then wade through, vs a model populating text closer to my reading speed.
    It would be interesting to pair the super fast model with a normal speed model. Have the super fast one do all the background research, code writing, etc. The normal model would just relay the needed info to you at a more reasonable pace.
    drewstiff · 1790937523 · focus · HN ↗
    Why do you even need a second model for that? There are many simpler ways to slow down the rate that the near-immediate response is pseudo-typed onto the screen.

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.