Thread

Decision models like Jev don't beat LLM-as-a-judge or traditional classifiers

129 points · 46 comments · tomncooper

segmondy · 1790969354 · focus · HN ↗
duh, this is not news. (general, fast and cheap) before decision models, you could pick only 2.
LLM as judges - generalized, but too slow. If you had to make millions of classifications a day, this will be the wrong approach. you won't/shouldn't use LLM to classify spam/no spam. hot dog/or something.
traditional classifiers, very specific 1 trick pony, super fast and cheap once built. If you need to make tons and tons of classifications, this would be the approach. but if you wanted a classifier right now for a novel problem, you need an expert to curate data, train and deploy.
decision models/jev - are generic, you can throw them at most generic classification problems, and they are good enough. it's a fine balance between general, fast and cheap. you get all 3
1. lostmsu · 1790976889 · focus · HN ↗
  Is Jev any faster, cheaper, or more precise than medium LLMs like Luna 6?
  1. ShinTakuya · 1791189152 · focus · HN ↗
    Yes. Not as fast/cheap as Typesafe claims, but lots of benchmarks suggest around 3-4 times cheaper, and 7-14 times faster.
    - <a href="https://www.ml6.eu/en/blog/jev-vs-gpt-6-luna-vs-bert-text-classification" rel="nofollow">https://www.ml6.eu/en/blog/jev-vs-gpt-6-luna-vs-bert-text-cl... - <a href="https://tessl.io/blog/jev-is-136x-faster-and-27x-cheaper-than-gpt-luna-6-for-tessl-verifiers-try-it-yourself" rel="nofollow">https://tessl.io/blog/jev-is-136x-faster-and-27x-cheaper-tha... - <a href="https://x.com/fazxes/status/2100300097695232164" rel="nofollow">https://x.com/fazxes/status/2100300097695232164 (this last one is Luna 5.6 but that isn't too different from 6 besides accuracy and cost)
    saberience · 1791208136 · focus · HN ↗
    No shit its cheaper and faster, thats the case with every small model.
    It's a trade off between wanting something dumber but fast and cheap, or something slower and more expensive that is vastly more capable and smarter.
    andy12_ · 1791208717 · focus · HN ↗
    But the thing is that across a lot of classification benchmarks, Jev is actually faster, cheaper, and smarter than Luna (at least without reasoning). For example, look at this one from another user in this thread [1]
    [1] <a href="https://hard-decisions.anth.us/models/" rel="nofollow">https://hard-decisions.anth.us/models/

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.