Thread

1. AgentMasterRace · 1790631907 · focus · HN ↗

   I compared it to Jev in my current use cases and it's very inaccurate. 70% vs 94% . for classification, it's unacceptable.

   1. tbeseda · 1790633130 · focus · HN ↗

      For _your_ classification it's unacceptable. The OP seems to have anticipated this and mentions you can fine tune it for your use case. Did you try that?

      I don't think the point is to displace Jev, but to show it's possible to build an MVP on open weights without years of work and millions of dollars.

      Why (presumably) an engineer would dismiss exploring a lightweight, custom alternative to locking into a fashionable PaaS, I'll never know.

      1. senko · 1790636014 · focus · HN ↗

         > The OP seems to have anticipated this and mentions you can fine tune it for your use case. Did you try that?

         You can already so that with classification models such as ModernBERT, at 0.4B.

         Jev's value is its zero shot performance without having to fine-tune.

         1. beepbooptheory · 1790641253 · focus · HN ↗

            I am sure I am missing something obvious here, but why is that valuable? Like, what kinds of projects are there where you need to classify stuff but are unable to make a bespoke model targeting the specific problem?

            1. shaewest · 1790642225 · focus · HN ↗

               For my org, it meant we could trial classifiers across various internal systems with little to no engineering effort. In one case we ended up building our own classifier instead of Jev, but in others we kept Jev because it was zero-effort for a great impact.

               1. beepbooptheory · 1790650763 · focus · HN ↗

                  But like how often are you gonna do this in general? Why does dev time or effort really matter here when either way you are building something to just, you know, actually use going forward? Its not like one needs to build a new classifier everyday.

                  1. tyre · 1790691446 · focus · HN ↗

                     Because a lot of people employed as engineers can’t build classifiers. They’re able to glue libraries and tools together, build UIs, and write APIs, but don’t have the curiosity or creative problem solving to learn and master a new domain. Even when “master” is scoped to something like this.

                     On top, most EMs wouldn’t take a risk on an exploration of something “unknown” (to them) and couldn’t get buy-in from a PM.

                     I say this as an EM. Interview hundreds of people and, while, yes, some people don’t interview well, you might be shocked at the level of creative thinking. Even when “creative” is narrowly scoped to “this is a solved problem in a related domain.

                     1. beepbooptheory · 1790701525 · focus · HN ↗

                        I guess this makes sense. I am no business guy, but if my product/company was focused on some sort of classification problem, my naive intuition would be to focus on hiring guys that can do it, rather than try to make the problem easier for them. But perhaps at the end of the day this is still cheaper? It's the same reason why we use Postgres instead of hire database experts to build something special?

                        1. senko · 1790716625 · focus · HN ↗

                           > if my product/company was focused on some sort of classification problem

                           It's more likely the product is focused on something else, but a classification model could come in handy...

Unofficial Hacker News client; not affiliated with Y Combinator.