Thread

Beating GPT-5.6 Sol on retrieval with 100x cheaper open models

414 points · 114 comments · moonikakiss

breadislove · 1785959360 · focus · HN ↗
On what do you guys test the model. Its very dubious that there is no common retrieval benchmark such as browsecomp plus or similar tested. And what metric do you report?
1. krm01 · 1785960775 · focus · HN ↗
  Keeping track of any AI progress is becoming harder by the day, because there's ambiguity around common/clear/consistent benchmarks. Everything is constantly skewed into favourable directions.
  1. alansaber · 1785969427 · focus · HN ↗
    TBF gaming benchmarks is not something new to AI
2. seahyinghang8 · 1785981607 · focus · HN ↗
  (founder of castform here) - we didn't get to dive too deep into the dataset we were using for the retrieval in the blogpost for brevity, but we did link the training run (which shows the dataset) here: <a href="https://app.castform.com/train/a7a898f6-d802-4908-b044-acb812f14a48?tab=comp" rel="nofollow">https://app.castform.com/train/a7a898f6-d802-4908-b044-acb81...
  the page shows the exact trace of all the models we are comparing against and the aggregate scores
  we generated the question & answer pair from gitlab product handbook (<a href="https://handbook.gitlab.com/" rel="nofollow">https://handbook.gitlab.com/) since the point is to show that you can generate training questions from raw data corpus (something a company already has today)

Unofficial Hacker News client; not affiliated with Y Combinator.