‹ BackHN Continuity

Thread

Jeff – Jev-compatible 0.8B decision models, trained at home, ~30 ms

575 points · 225 comments · firelex

  1. adrithmetiqa · · focus · HN ↗
    Forgive my lack of understanding but how long before Jev type functionality is just built straight into all frontier models?
    1. jubilanti · · focus · HN ↗
      Do people really have zero awareness that Structured Outputs with a constrained schema has been a thing for a while now, and open weight models that give you logprobs can give you distributions per key?

      Like what am I missing? I use Structured Outputs every day and this just seems like that with fewer steps?

      edit: Where I'm coming from, I can triage 10,000 support tickets with deepseek flash for less than $1, and latency is sub 1 second if it needs to be integrated into a live user flow. I don't need anything cheaper or faster than that.

      1. fzysingularity · · focus · HN ↗
        FWIW you’re absolutely correct on how most developers are unaware of this as they’re mostly operating on the API level and not aware of server side (vLLM/SGLang) capabilities.

        The aspect that I like the most is the typesafe API that introduces new probabilistic concepts that are more sound than json schema and constrained decoding with quasi-confidence scores. Developers were asking LLMs to also emit confidences which made absolutely no sense whatsoever.

        1. tclancy · · focus · HN ↗
          Can you link to some info on this? I am just starting to poke around in the space and would love a leg up.
          1. fzysingularity · · focus · HN ↗
            I don’t know of one definitive resource but “constrained decoding”, regex/grammar-based decoding, json-schema decoding will give you a bunch of hits. Look up vLLM, SGLang and outlines’ implementations for more technical details.

            This looks pretty decent: <a href="https:&#x2F;&#x2F;www.aidancooper.co.uk&#x2F;constrained-decoding&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.aidancooper.co.uk&#x2F;constrained-decoding&#x2F;

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.