I am hearing about Jev for the first time here so no idea about the hype.
So their(Jev) is that the thing is faster at classification than a frontier model? Because the whole type safe aspect is already fully solvable with structured output.
But their example is classification but that would also be possible and faster with a classic BERT model.
So their pitch is a task specific smaller model or am I completely misunderstanding the whole thing?
There's still run-to-run variance because it's not fully deterministic. So runs with exact same inputs can return different outputs. Besides, though the output always conforms to the choices you specified, whether the probabilities attached to them are actually correct is a different issue.
This seems like an unusual definition of type safety. I certainly understand how every run deterministically giving the same schema (type) of data is a requirement to be "type-safe", but in my mind the content of the result is not relevant to the question of type safety. Am I not getting it?
Jev HAS hallucinations and it doesn't attempt to solve hallucination at all.
For three choices problem (A,B,C), what Jev guarantees is that it will give the choice in a defined schema (type-safe). It never guarantees that the choice is correct (hallucination).
bruhhhhhh · · focus · HN ↗
idz · · focus · HN ↗
Not particularly. There is still the problem of hallucinations and varying results across runs.
That's more of what type-safety means for their team. Every run gives the same results. It's type-safe
kantahayashi · · focus · HN ↗
sanderjd · · focus · HN ↗
pasteleft · · focus · HN ↗
For three choices problem (A,B,C), what Jev guarantees is that it will give the choice in a defined schema (type-safe). It never guarantees that the choice is correct (hallucination).