Thread

1. netvarun · 1790533887 · focus · HN ↗

   Off topic:With sol pricing drop tbh kimi k3’s value prop has not been that great. For our internal use case/testing/benchmarks sol come out with way better quality and much cheaper costs. Kimi really needs to drop their pricing (I heard it’s set by them across all the neoclouds) Sol is at 2/10 vs kimi’s 3/15

   1. nicce · 1790540750 · focus · HN ↗

      Sol pricing dropped but so did the quality few days ago. I wonder when these companies are sued for making the terms from their side to go downwards while taking the same subscription cost.

      1. solarkraft · 1790540945 · focus · HN ↗

         Is anybody tracking these quality changes? All I've seen so far are accusations (quite a few at this point) but not really any actual data.

         1. pupppet · 1790541524 · focus · HN ↗

            I don’t understand how there isn’t a website out there tracking this stuff already.

            1. bpavuk · 1790542934 · focus · HN ↗

               how the hell do we even track that? and before someone says...

               —"Benchmarks!"

               ...I'll tell that they can be gamed so easily, and they are on a consistent basis.

               1. solarkraft · 1790547083 · focus · HN ↗

                  Sure they are, but do you think they are continuing training to improve a model after release without bumping the version number, presumably only to game the benchmarks?

                  1. nananana9 · 1790566908 · focus · HN ↗

                     If you're willing to cheat, isn't it just a matter of grepping for the benchmark's question and pasting the solution in the chain of thought?
                  2. conorcleary · 1790621163 · focus · HN ↗

                     like windows 9
            2. copperx · 1790551331 · focus · HN ↗

               <a href="https:&#x2F;&#x2F;modelregression.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;modelregression.com&#x2F;

               1. hn8726 · 1790554915 · focus · HN ↗

                  Haven&#x27;t looked into how accurate the page is, but the list of regressions on the bottom looks terrifying, at first glance?

                  1. copperx · 1790555731 · focus · HN ↗

                     Yes, it looks like regressions are frequent, but sometimes performance goes back to baseline quite fast.

Unofficial Hacker News client; not affiliated with Y Combinator.