Thread

Clef: Open-weight decision models, and new RL fine-tuning platform

637 points · 217 comments · jasondavies

manlymuppet · 1790875177 · focus · HN ↗
Am I hearing this right, that they made a decision model based on Typesafe's new paradigm, and actually made a model better than Jev based on Typesafe's own ranking?
And it's only been a few weeks.
1. TeMPOraL · 1790876066 · focus · HN ↗
  It's not a "new paradigm", it's a low-hanging fruit that's been lying around for years; Typesafe were the first to bother to stop and pick it up, and market the shit out of it. But it was still a low-hanging fruit.
  There are many, many of those left around, because AI frontier is moving forward so fast, everyone is racing ahead. Which is why I laugh when people say AI is not transformative and LLMs are a dead end (and my favorite, "what are we going to do with all those GPUs when the bubble pops?"). Even if SOTA LLMs hit a hard capability limit tomorrow and never advanced again, there's a good decade of growth and advancement to be extracted just from all the low-hanging fruits that were left unpicked along the way.
  1. seizethecheese · 1790876112 · focus · HN ↗
    Name a few of these low hanging fruit left around.
    murkt · 1790876275 · focus · HN ↗
    Easy to reach doesn’t automatically mean “easy to see”.
    [deleted] · 1790876467 · focus · HN ↗
    [deleted]
    TeMPOraL · 1790876903 · focus · HN ↗
    Jev is one.
    Diffusion transformers are not "easy" but underfunded.
    Random one in terms of applications: getting GPT-4-level[0] LLMs to operate at hundreds of tokens per second on edge hardware - opens up so many possibilities I'm probably unable to imagine half of them.
    E.g. Imagine spellcheck/predictive text (or code autocomplete) where the model is able to process a whole paragraph + surrounding application/system context in between keystrokes. Or an OS being able to reliably guess what you're doing in real-time, in between your UI interactions, and offer actually helpful contextual reactions.
    Or imagine finally funding some decent studies into exploring the models as computational artifacts - studying their latent spaces, how they form and how they model reality internally.
    Or imagine automated sliding doors that don't suck.
    --
    [0] - Or anything substantially better than BERT-level models used in Jev or that demo from the company doing inference ASICs, that has a chatbot online that does 14 kilotokens per second.
    aeve890 · 1790877840 · focus · HN ↗
    >Random one in terms of applications: getting GPT-4-level[0] LLMs to operate at hundreds of tokens per second on edge hardware
    That's low hanging for you?
    blurbleblurble · 1790878356 · focus · HN ↗
    It's likely quite close. There are so many papers proving concepts that would bring this, they just haven't been combined in production.
    msdz · 1790878366 · focus · HN ↗
    Maybe they meant in the sense of “untapped potential”, because so far a lot of the focus has been on increasing model capabilities, not necessarily performance/power budget.
    TeMPOraL · 1790880844 · focus · HN ↗
    Yes. Point is, it's untapped only because everyone is running in the race (even if out of curiosity), and there's just not enough people with means to tap into these side threads. For the past few years, there's been many interesting papers that circulated the industry, got recognized as worthwhile pursuits, and then dropped because running behind the Big Vendors had massively better ROI.
    ekabod · 1790878678 · focus · HN ↗
    That's a high hanging fruit, not low.
    guyomes · 1790879773 · focus · HN ↗
    If we throw in hardware dedicated to a specific LLM, it seems to be a rather low hanging fruit. Especially considering that this is already happening for vision models [1].
    [1]: "FPGA-based CNN Acceleration using Pattern-Aware Pruning" <a href="https://inria.hal.science/hal-04689673/document" rel="nofollow">https://inria.hal.science/hal-04689673/document
    mdp2021 · 1790886835 · focus · HN ↗
    > hardware dedicated to a specific LLM
    That wording screams "Taalas". Which, importantly, is not the only player trying to abate the distance between data and arithmetics...
    TeMPOraL · 1790880673 · focus · HN ↗
    Yes. It's well within realm of possibility, but so far wasn't pursued because the Big Vendors went all-in into capability growth (rightfully testing "the bitter lesson" to its limits) and got themselves stuck in an arms race, while everyone else is barely keeping up and/or starstruck with fascination, exploring what these models can do.
    This got everyone racing forward and right now there is not enough human attention left in the world to productionize this, or any of the other "side threads". When the race slows down, people will catch up, branch out, and loop back.
    blurbleblurble · 1790881018 · focus · HN ↗
    Just like renewable energy and so many other things. Hyperconcentration of capital is really tragic. I hope things turn around.
    TeMPOraL · 1790881569 · focus · HN ↗
    They will. That's the fallacy of the "S-curve" everyone likes to commit these days actually giving a positive outlook.
    Assuming it won't get to full RSI, the current approach will burn out - most likely economically. The race slows down, people branch out, look back, start picking up the "untapped potential"/low-hanging fruits, and you have new S-curves launching in place of the one that just tapered off (hence a fallacy - a stack of S-curves adds up to continuing exponential growth).
    In other words: it comes and goes. Hyperconcentrated capital will eventually deconcentrate.
    mdp2021 · 1790884152 · focus · HN ↗
    > That's low hanging for you
    An important part of the industry is studying that: it is built-up effort. Sooner or later, the fruits will be harvested. The targeted preparation has been there for years now.
    Twirrim · 1790890577 · focus · HN ↗
    We already have examples of LLMs running 16k+ tokens a second using custom ASICs.
    It's down at the moment (Not sure if it'll return?) but Chat Jimmy[0] produced by Taalas[1] was powered by an ASIC running Llama 3.1 8B, and hitting 17,000 tokens/sec. It was amazing to use, you'd no sooner have hit enter than you had a full response back. I actually found its speed to be a problem for interactive stuff, as every answer got several paragraphs I'd then wade through, vs a model populating text closer to my reading speed.
    I appreciate, there are differences between an 8 billion parameter model and something GPT-4-ish, but we're currently in the middle of a race between a half dozen or so companies to produce the next best frontier model, which requires their infrastructure to be dynamic.
    We really don't always need newer better faster stronger models, there's quite a lot of room for "good enough" where getting 17kt/s at significantly lower power would be amazing.
    [0] <a href="https://chatjimmy.ai/" rel="nofollow">https://chatjimmy.ai/ [1] <a href="https://taalas.com/" rel="nofollow">https://taalas.com/
    AshamedBadger56 · 1790891575 · focus · HN ↗
    >I actually found its speed to be a problem for interactive stuff, as every answer got several paragraphs I'd then wade through, vs a model populating text closer to my reading speed.
    It would be interesting to pair the super fast model with a normal speed model. Have the super fast one do all the background research, code writing, etc. The normal model would just relay the needed info to you at a more reasonable pace.
    drewstiff · 1790937523 · focus · HN ↗
    Why do you even need a second model for that? There are many simpler ways to slow down the rate that the near-immediate response is pseudo-typed onto the screen.
    flipping_beacon · 1790877965 · focus · HN ↗
    Definitely agree with edge computation, although inference extensively researched and funded if SOTA LLMs hit a dead end tomorrow,there is still a lot to explore and research in inference and edge computation
    blurbleblurble · 1790878285 · focus · HN ↗
    Diffusion models combined with these new looping techniques are gonna change the whole conversation about efficiency. Imagine control net but in one or more conceptual latent spaces.
    But also harnesses and more generally new insights on "the control flow problem" could end up squeezing a ton of performance out of small models.
    Amekedl · 1790880252 · focus · HN ↗
    yeah your reply, nobody can predict the future.
    Enough stuff can happen, software use itself might change, and that could really cause anything. "What will we do with all the gpus" might become a question if for a magnitude of tech and reasons leaked-opus-9 runs on a macbook m6 or 7
    seizethecheese · 1790883122 · focus · HN ↗
    I commend you for actually answering, independent of what I think of the answers.
    dominotw · 1790886537 · focus · HN ↗
    not a very good answer though
    tomrod · 1790914018 · focus · HN ↗
    Yes yes, most comments and responses fail to contribute to conversation (though I disagree with your particular assessment); it thus remains important to continue to share our own thoughts as we battle through thesis and antithesis to arrive at synthesis.
    alightsoul · 1790883661 · focus · HN ↗
    [dead]
    mdp2021 · 1790883767 · focus · HN ↗
    There are "low hanging fruits" - easier to achieve goals -, and there are super-fruits, milestone-fruits.
    Among the most important ones:
    -- the long-known Problem of Transparency, applied to the apparent emergent intelligence in NNs. Why does it happen - in detail?
    -- then, a Theory of Apparent Intelligence through NNs. Transforming the results achieved into a Science. Which allows to do what we are doing - but in a lean and targeted way.
    -- then, a General Theory of Intelligence, that includes the above to go beyond current architectures and get those features of Intelligence we expect and still not have.
    The long-term direction we got into must lead to this.
    (You note a ponderant detail of the above when you note the importance of explaining the emergence of a World Model from a Language Model.)
    TeMPOraL · 1790885506 · focus · HN ↗
    Those are the absolutely fascinating parts, and I sincerely hope AI won't get out of control before we're able to tackle some of these.
    patcon · 1790887854 · focus · HN ↗
    If anyone is interested, following Dr Michael Levin's Thoughtforms.life podcast is the cutting edge of where all this previously fuzzy stuff is becoming more concrete. So long as you can tolerate distinguished scientists flailing about as they discuss consciousness and life and developmental biology (and other less-obviously living things, like algorithms) as involving "free lunches" and "ingressing patterns from the platonic realm" :)
    hobofan · 1790877285 · focus · HN ↗
    Closely connected to decision models: A good library to do ranking based on pairwise ranking on multiple attributes. By using a decision model (especially one that can make decisions on multiple fields at the same time) this becomes a lot faster and more powerful. Could make for a pretty nice search reranker as well as prioritizer for many problems.
    Of course you can also do ranking one-off with a decision model, but this likely less stable, and by doing pairwise ranking you can also relatively quickly do incremental inserts to the list.
    sarkarghya · 1790881821 · focus · HN ↗
    I can imagine advancements on making smaller models work together better instead of a generalized core. Imagine a community or city having a <a href="https://pirateface.co/" rel="nofollow">https://pirateface.co/ so that the shard of the model that you need can be streamed in with minimal latency with your box only holding the minimal version (say deepseek v4 flash as orchestrator) of the model that you use on day to day basis.
    We have overcome split brain problems before so this wont be our first
    esseph · 1790882829 · focus · HN ↗
    You're putting a lot of trust in uncorrupted, untrusted, unknown models (potentially).
    CamperBob2 · 1790883830 · focus · HN ↗
    An example I like to use is: compare the quality and scope of games released with a brand-new console to the ones released for that console towards the end of its life, when everyone has learned how to take advantage of whatever weird, wacky hardware Sony invented for that console generation.
    There is still a lot we don't know about how to get the most out of existing LLM components from a speed or cognitive-performance perspective. People could easily spend the next decade studying and refining what's been built so far, even if no new, original approaches ever arrive.
    randomNumber7 · 1790943760 · focus · HN ↗
    I never enjoyed playing ps4 when it is constantly louder than my vacuum cleaner.
    sroussey · 1790891606 · focus · HN ↗
    Just look at all the model type on hugging face. LLMs are a small percentage.
  2. btown · 1790893785 · focus · HN ↗
    Synthetic data is key here! Compared to 5 years ago, we now have oracles that can generate massive data sets of perfectly labeled multimodal data, practically for free. The number of architectures that can benefit from that is innumerable, and far beyond just LLMs themselves. On top of this, LLMs can implement any architectural ideas you have, and write custom tools to manage training and evaluation.
    Whether or not LLMs can self-improve their frontier capabilities, they can absolutely create a wake for themselves that accelerates everything else that's training on their synthetic data. We'll see every architecture of the past 40 years suddenly show leaps and bounds.
    gutchapa · 1791058992 · focus · HN ↗
    Agreed on the directions, it might appear to be piece of cake, but the "practically for free" part hides the hard bit: realism. Try simulating flight-search data with source, destination, flight numbers, routes, schedules , try a diff it against a real corpus. Even frontier models produce data that looks right at first sight, but breaks on joint distributions (a flight number that never flies that route, layovers that violate minimum connect times). last mile is the daunting task.
    btown · 1791061873 · focus · HN ↗
    To lean into your example: if you want to design travel search, you might envision a system that goes from a user's typed query -> a structured query -> executed against a database of real-time flights -> judging the result for feasibility of the resulting proposals created by those queries.
    What you can do with LLMs is reverse this: from any numbers of snapshots of flight data, you can create large numbers of plausible user queries, based on your data, that are known to be feasible or infeasible. And now you have a labeled data set to train a model that focuses solely on the query-creation and judgment systems. And you can experiment with whether having a more flexible query protocol leads to higher success rates without sacrificing accuracy, or whether you can generate that last-mile feasibility check as a combination of auditable code checks alongside AI-based judgment.
    LLMs don't absolve you of having to break down your system architectures into components that have well-defined boundaries (though certainly they can help with that design). They do make those components feasible to solve at scale.
  3. dzonga · 1790900628 · focus · HN ↗
    my take there's a lot of low hanging fruit in applying small models to knowledge economy workflows.
    then vision & robotics.
    while everyone's chasing the frontier.
  4. outofpaper · 1790924375 · focus · HN ↗
    Yup, one output token, and read the logprobs of the posible tokens that could have been thos 1st token. Plenty of systems already do this; Typesafe's fundraising and marketing just made it visible.
    Good to see interest broadening beyond "just extend thinking." More approaches in the toolbox means fewer problems get treated as nails.
  5. tomrod · 1790979454 · focus · HN ↗
    > what are we going to do with all those GPUs when the bubble pops?
    Your comment here made me laugh, because I think we will finally be able to play Crysis at 10fps.
    Just kidding, of course. GPU half lives are quite a bit less than standard compute half lives, no?[0] That's what I've been trying to understand regarding data centers focusing as GPU clusters -- seems like the ROI window would have to be very short for the capitalization.
    [0] <a href="https://www.tomshardware.com/pc-components/gpus/datacenter-gpu-service-life-can-be-surprisingly-short-only-one-to-three-years-is-expected-according-to-unnamed-google-architect" rel="nofollow">https://www.tomshardware.com/pc-components/gpus/datacenter-g...
2. slopnt · 1790876136 · focus · HN ↗
  They have to have decision models already in production. Part of their business is detecting bots, DDoSers and spammers.
  1. alightsoul · 1790883591 · focus · HN ↗
    Yeah that's a decision tree, random Forest or some other machine learning classifier. They have existed for a long time
    smallmancontrov · 1790884108 · focus · HN ↗
    I'm all for rebranding discriminative models as decision models, though.
    "Discriminative" always had pointlessly bad optics, but I knew it was over when I started seeing prominent machine learning researchers who p=100% knew better describe discriminative models as generative because that was the buzzword of the year. "Decision model" sells the value proposition much better and doesn't sound like an anti-woke crusade.
    tomrod · 1790913783 · focus · HN ↗
    Decision models have long been called `choice` models to may understanding. Huge statistical literature on dynamic discrete choice models and inference!
    eastdakota · 1790904145 · focus · HN ↗
    I think you just called me old.
3. segmondy · 1790890663 · focus · HN ↗
  A lot of people claim to have made better than jev, there's a jev benchmark, I have tried many of those models and they eventually end up failing, a non trivial task which doesn't seem like much but reminds me of the svg pelican bench is games, have one of these decision/classifier models play a game, hook it up to the input, most of the ones that are supposedly on jev level end up playing a terrible game, showing that they are very narrow. Cloudflare doesn't compare to the top open bench alternatives, I just finished downloading it and will compare it to jev for non trivial tasks tonight.
  1. SebastianSosa · 1790894835 · focus · HN ↗
    Public benchmarks are easy to cheat, if I am typesafe I would also release a public benchmark to distract otherwise competent people in overfitting to a benchmark instead of making something actually useful. Diogo very much is against public benchmarks ;)
    Foobar8568 · 1790942920 · focus · HN ↗
    You take Qwen3.6 35b on a 5090rtx, and here you get a higher score than Jev, for 2sec more latency on average. So yeah it's not subsecond, but I am sure that if I had VC money, I could too get within 500ms too!
  2. verdverm · 1790894963 · focus · HN ↗
    watching Jev play Pokemon demonstrated this too, more hype than meat
    - I'd like a potion, are you sure, no, repeat
    - in and out of doors on loop
    - sisyphean effort in the cave
    - jev-ish level grinding
    It was impressive, beat pokemon for less than $2, but not all that interesting. People asking how different Math.Random plays pokemon would be, and at the other end, regular llms playing games.
  3. indoor47 · 1790939958 · focus · HN ↗
    Well, it makes sense:
    "Clef builds upon this concept, but uses a different base model as the backbone. We currently use Qwen as the base model and post-trained it to suit decision model use cases. "
4. heliosAtwork · 1790898223 · focus · HN ↗
  There was some parallel independent work from Sep 28. They mention it in the blog post. But I am sure Jev has opened a lot of eyes on the possibilities.
  "In the same week that Jev came out, we posted about some experiments [1] we had with our own homegrown decision model."
  [1] <a href="https://x.com/michellechen/status/2101091012559151480" rel="nofollow">https://x.com/michellechen/status/2101091012559151480
5. lofaszvanitt · 1790899369 · focus · HN ↗
  Race to the bottom. The one with the most resources wins.
6. fwip · 1790944537 · focus · HN ↗
  From their blog post, it sounds like it is still 10x as slow as Jev.
  As far as I've seen, all of the Jev-compatible projects simply take an LLM, hack off a layer or two at the end, and call it good. Some of them spend more work than others trying to back-estimate in accurate probabilities.
  1. rahimnathwani · 1790958946 · focus · HN ↗
    Yeah I think people were already using either of these as part of traditional software workflows):
    A) Structured outputs from LLMs (doesn't need fine tuning but can be expensive)
    B) Classification output from fine-tuned BERT-like or GLiNER models (is calibrated well and has cheap/fast inference)
    What Jev did is combine the advantages of both A and B into one model/product, and create a really good API.
    They claim that a key innovation is how they've trained the model using what they call RLCD (RL from calibrated decisions). So it's not just that you can get the outputs (which is easy to add to any LLM) but that the different primitives they expose (Choice, Score, Noul) have each been calibrated. For example, they claim that if you use the Score primitive (which gives you probabilities along a bunch of choices representing a continuum) that's not just using the more general 'Choice' primitive under the hood. It's been calibrated separately.
    I don't know how many of the Jev-like things we've seen do that. For example Cloudflare offers a Jev-like model with the same API, and which they say was trained with RLCD. But I don't know whether Choice and Score are different under the hood, or whether Score is just sugar on top of Choice. (Should be easy to test this, but I haven't done it.)
7. nater5000 · 1790948155 · focus · HN ↗
  >And it's only been a few weeks.
  You make it sound like this is some noteworthy timeframe. I'd be more surprised if Jev wasn't immediately made obsolete within days of release (which was effectively the case), especially when backed by a company like Cloudflare lol
  ML moves quick. Add the extreme hype and cash floating around in this space and you can expect that anything resembling something novel and relatively untapped is going to be pounced on and turned over basically immediately.
8. xnx · 1790961471 · focus · HN ↗
  What's the hype with Jev? Hasn't Gemini had structured outputs since November 2025? <a href="https://ai.google.dev/gemini-api/docs/structured-output" rel="nofollow">https://ai.google.dev/gemini-api/docs/structured-output
  1. manlymuppet · 1790970165 · focus · HN ↗
    Traditional LLMs doing structured output is like trying to fit a square peg in a round hole. It's just not the right tool for the job. They can do structured output, but it's janky.
    With Jev, structured output is its native format. Jev is just way faster, cheaper, and outright better for a lot of things.
    Note that while all of this is great, this is nowhere near a sort of "ChatGPT moment". It's a cool new thing, and it's way better at certain tasks, which could be big. That's all though.

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.