Thread

1. AgentMasterRace · 1790631907 · focus · HN ↗

   I compared it to Jev in my current use cases and it's very inaccurate. 70% vs 94% . for classification, it's unacceptable.

   1. tbeseda · 1790633130 · focus · HN ↗

      For _your_ classification it's unacceptable. The OP seems to have anticipated this and mentions you can fine tune it for your use case. Did you try that?

      I don't think the point is to displace Jev, but to show it's possible to build an MVP on open weights without years of work and millions of dollars.

      Why (presumably) an engineer would dismiss exploring a lightweight, custom alternative to locking into a fashionable PaaS, I'll never know.

      1. nico · 1790633859 · focus · HN ↗

         Not sure the task at hand here. But if it doesn’t require any reasoning/thinking and it’s just a classification task, it’s worth a shot to look into training your own classifier

         I’ve run some benchmarks. Using embeddings + logistic classifier, the architecture matches or beats Jev and Laya in all basic classification tasks (datasets tested: AG News, Emotion, MASSIVE Intent, Banking77) The type of task in which it does really well, especially against Laya, is classification with >50 classes

         The classifiers also run in <1ms, so they can be very fast and precise at the same time

         But this architecture has no “reasoning”, so it performs rather poorly on tasks that require it, like the ones from the XLNI dataset (Jev/Laya do a lot better on this one)

         For the latter cases, you could use add a local lightweight LLM, something like a Gemma model. Or even some basic MLP, depending on the tasks/data

         1. rahimnathwani · 1790637049 · focus · HN ↗

            "Using embeddings + logistic classifier, the architecture matches or beats Jev and Laya in all basic classification"

            Have I understood correctly that you trained only the logistic classifier, but didn't need to train the embedding model?

            If so, I'm curious whether you compared that approach (A) with:

            B) Jev only, with a single output.

            C) Jev with multiple outputs fed into a logistic classifier.

            Obviously C has cons (can't be self-hosted, needs some up-front work on deciding the shape of the output) but it might be somewhat more interpretable. (And I suppose it might have better performance?)

            1. nico · 1790639194 · focus · HN ↗

               You are correct, I didn’t train the embeddings model

               Here&#x27;s a gist with code you can use to test the Banking77 dataset: <a href="https:&#x2F;&#x2F;gist.github.com&#x2F;nicobrenner&#x2F;056a5aaff5d0119c0032ecdad5029557" rel="nofollow">https:&#x2F;&#x2F;gist.github.com&#x2F;nicobrenner&#x2F;056a5aaff5d0119c0032ecda...

               The gist uses BAAI&#x2F;bge-large-en-v1.5, which is 1.2GB approx. You can replace it for all-MiniLM-L6-v2 (91 MB @ fp32 or 45 MB quantized fp16) small enough for mobile&#x2F;edge. With all-MiniLM-L6-v2 it still gets 93.0% on Banking77, only 1.3 points behind bge-large at 15x smaller

               I haven’t compared different ways of sending requests to Jev

               The data to train the classifiers comes from the datasets used to test them (not from Jev)

Unofficial Hacker News client; not affiliated with Y Combinator.