Thread

Beating GPT-5.6 Sol on retrieval with 100x cheaper open models

414 points · 114 comments · moonikakiss

aliljet · 1785957209 · focus · HN ↗
There is a more serious question in here that's not being answered. How effective is the retrieval in finding buried needles in larger and larger haystacks. And there's a correlary question, how effective could you be in finding paired needles in that haystack where you need to hold a needle to unlock finding another needle.
1. Foobar8568 · 1785961985 · focus · HN ↗
  Considering the state of the field ( RAG/retrieval/evaluation) I have 0 trust in it, even more if it's closed source with bullshit claim like that.
  Everything is vibe sloped to death, and dead after a few months to a couple of years (and not hard to be 100 cheaper than GPT-5.6 sol ... DS is basically free and I guess already 100 times cheaper or more, and here another slope ).
  1. kumama · 1785981382 · focus · HN ↗
    castform founder here. we should have made it more prominent on the blog but here's the full code example: <a href="https://github.com/castform-ai/benchmax/tree/main/examples/neon_rag" rel="nofollow">https://github.com/castform-ai/benchmax/tree/main/examples/n...
    sreekanth850 · 1785988378 · focus · HN ↗
    Are you using blind chunking or section aware chunking?
    kumama · 1785989407 · focus · HN ↗
    for the example here the chunking is section aware -> but the general training data synthesis pipeline is agnostic to type of chunking
    sreekanth850 · 1785989924 · focus · HN ↗
    if chunking is section aware, how do you manage large section embeddings?

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.