Thread

Decision models like Jev don't beat LLM-as-a-judge or traditional classifiers

114 points · 43 comments · tomncooper

segmondy · 1790969354 · focus · HN ↗
duh, this is not news. (general, fast and cheap) before decision models, you could pick only 2.
LLM as judges - generalized, but too slow. If you had to make millions of classifications a day, this will be the wrong approach. you won't/shouldn't use LLM to classify spam/no spam. hot dog/or something.
traditional classifiers, very specific 1 trick pony, super fast and cheap once built. If you need to make tons and tons of classifications, this would be the approach. but if you wanted a classifier right now for a novel problem, you need an expert to curate data, train and deploy.
decision models/jev - are generic, you can throw them at most generic classification problems, and they are good enough. it's a fine balance between general, fast and cheap. you get all 3
1. lostmsu · 1790976889 · focus · HN ↗
  Is Jev any faster, cheaper, or more precise than medium LLMs like Luna 6?
  1. ShinTakuya · 1791189152 · focus · HN ↗
    Yes. Not as fast/cheap as Typesafe claims, but lots of benchmarks suggest around 3-4 times cheaper, and 7-14 times faster.
    - <a href="https://www.ml6.eu/en/blog/jev-vs-gpt-6-luna-vs-bert-text-classification" rel="nofollow">https://www.ml6.eu/en/blog/jev-vs-gpt-6-luna-vs-bert-text-cl... - <a href="https://tessl.io/blog/jev-is-136x-faster-and-27x-cheaper-than-gpt-luna-6-for-tessl-verifiers-try-it-yourself" rel="nofollow">https://tessl.io/blog/jev-is-136x-faster-and-27x-cheaper-tha... - <a href="https://x.com/fazxes/status/2100300097695232164" rel="nofollow">https://x.com/fazxes/status/2100300097695232164 (this last one is Luna 5.6 but that isn't too different from 6 besides accuracy and cost)
    [deleted] · 1791193183 · focus · HN ↗
    [deleted]

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.