Thread

Beating GPT-5.6 Sol on retrieval with 100x cheaper open models

414 points · 114 comments · moonikakiss

aliljet · 1785957209 · focus · HN ↗
There is a more serious question in here that's not being answered. How effective is the retrieval in finding buried needles in larger and larger haystacks. And there's a correlary question, how effective could you be in finding paired needles in that haystack where you need to hold a needle to unlock finding another needle.
1. Foobar8568 · 1785961985 · focus · HN ↗
  Considering the state of the field ( RAG/retrieval/evaluation) I have 0 trust in it, even more if it's closed source with bullshit claim like that.
  Everything is vibe sloped to death, and dead after a few months to a couple of years (and not hard to be 100 cheaper than GPT-5.6 sol ... DS is basically free and I guess already 100 times cheaper or more, and here another slope ).
  1. kumama · 1785981382 · focus · HN ↗
    castform founder here. we should have made it more prominent on the blog but here's the full code example: <a href="https://github.com/castform-ai/benchmax/tree/main/examples/neon_rag" rel="nofollow">https://github.com/castform-ai/benchmax/tree/main/examples/n...
    sreekanth850 · 1785988378 · focus · HN ↗
    Are you using blind chunking or section aware chunking?
    kumama · 1785989407 · focus · HN ↗
    for the example here the chunking is section aware -> but the general training data synthesis pipeline is agnostic to type of chunking
    sreekanth850 · 1785989924 · focus · HN ↗
    if chunking is section aware, how do you manage large section embeddings?
    Foobar8568 · 1785990265 · focus · HN ↗
    Thank you for the github link, what is the license? I went briefly over several repo, unless I missed it somewhere the license was often missing.
    kumama · 1785990752 · focus · HN ↗
    it's apache 2.0: <a href="https://github.com/castform-ai/benchmax/tree/main?tab=Apache-2.0-1-ov-file" rel="nofollow">https://github.com/castform-ai/benchmax/tree/main?tab=Apache...
2. BikiniPrince · 1785964159 · focus · HN ↗
  I use detailed project files. It has data regarding the project and subtasks as well as task status. It doesn’t depend on agent context and it’s managed to keep the agent on track. Feature creep with the new models is a very real issue. Capturing principles and how to reconcile tasks helps too. Even today it brought up a source of truth issue it had detected. There were multiple authorities born out of a patch and it used that principle to highlight and resolve the problem.
3. seahyinghang8 · 1785982743 · focus · HN ↗
  <founder of castform here> tldr: we generated synthetic training questions from the gitlab product handbook.
  totally agree that this larger corpus with harder to search information would be a good way to stress test - i'm sure we will encounter more interesting problems to solve. love to hear any suggestions of corpus to search against that is not just the public internet
  the training run link is also a little buried but here, you can see the comparison against the various models and their exact traces: <a href="https://app.castform.com/train/a7a898f6-d802-4908-b044-acb812f14a48?tab=comp" rel="nofollow">https://app.castform.com/train/a7a898f6-d802-4908-b044-acb81...

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.