Thread

1. adrithmetiqa · 1790632786 · focus · HN ↗

   Forgive my lack of understanding but how long before Jev type functionality is just built straight into all frontier models?

   1. wgd · 1790641609 · focus · HN ↗

      Negative three years, give or take. Although recent Anthropic and OpenAI models no longer expose the capability. But for any open model you just tell it to respond with a single token "Y/N" and take the logit difference. If you want multiple distinct questions answered you just ask them independently and put the shared context first so it gets cached.

      OpenAI and Anthropic don't want to give out logprobs these days but could trivially add a dedicated classification API to their existing models if there was enough demand.

      1. Renaud · 1790641964 · focus · HN ↗

         I think the main differentiator offered by Jev is not the ability to answer questions, most models can be coerced into that function if they don’t already have a dedicated pipeline for it, rather it’s the extreme speed of the evaluation, and very low cost that opens new possibilities.

         1. wgd · 1790642062 · focus · HN ↗

            The evaluation is fast because it's all prefill computation with only a single token of inference. Ditto cost, you're paying 100% input costs and nearly zero output. There really isn't any architectural magic to Jev, it's just a straightforward application of normal LLM tech with some good marketing.

            I mean, Jev is also probably cheaper because it's a rather small model (or at least, I suspect it is based on the overall level of intelligence it demonstrates) so that helps make it cheap too.

            1. psyphy2 · 1790642571 · focus · HN ↗

               Yes this is exactly my assumption as well. I think ppl forgot before agents it was expected slo to have a ttft in range of a few hundard ms, which is what jev is also achieving.

               Thats also why it feels weird they say they don't "charge for output tokens" since its literally generating a single (or at most very few tokens).

         2. selcuka · 1790643344 · focus · HN ↗

            > it’s the extreme speed of the evaluation, and very low cost that opens new possibilities.

            It also returns confidence scores for all choices.

            Granted, they are not stable. They fluctuate even when you reorder choices, but it still counts as an additional feature.

            1. wonnage · 1790645099 · focus · HN ↗

               They’re useable if you’re trying to rank autosuggestions or something, not great at implying actual understanding

Unofficial Hacker News client; not affiliated with Y Combinator.