Thread

Beating GPT-5.6 Sol on retrieval with 100x cheaper open models

414 points · 114 comments · moonikakiss

aliljet · 1785957209 · focus · HN ↗
There is a more serious question in here that's not being answered. How effective is the retrieval in finding buried needles in larger and larger haystacks. And there's a correlary question, how effective could you be in finding paired needles in that haystack where you need to hold a needle to unlock finding another needle.
1. Foobar8568 · 1785961985 · focus · HN ↗
  Considering the state of the field ( RAG/retrieval/evaluation) I have 0 trust in it, even more if it's closed source with bullshit claim like that.
  Everything is vibe sloped to death, and dead after a few months to a couple of years (and not hard to be 100 cheaper than GPT-5.6 sol ... DS is basically free and I guess already 100 times cheaper or more, and here another slope ).
  1. kumama · 1785981382 · focus · HN ↗
    castform founder here. we should have made it more prominent on the blog but here's the full code example: <a href="https://github.com/castform-ai/benchmax/tree/main/examples/neon_rag" rel="nofollow">https://github.com/castform-ai/benchmax/tree/main/examples/n...
    Foobar8568 · 1785990265 · focus · HN ↗
    Thank you for the github link, what is the license? I went briefly over several repo, unless I missed it somewhere the license was often missing.
    kumama · 1785990752 · focus · HN ↗
    it's apache 2.0: <a href="https://github.com/castform-ai/benchmax/tree/main?tab=Apache-2.0-1-ov-file" rel="nofollow">https://github.com/castform-ai/benchmax/tree/main?tab=Apache...

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.