Thread

1. Chance-Device · 1789652006 · focus · HN ↗

   Pretty interesting, I’m sure it will be useful for anyone who is rolling their own RAG.

   1. entrope · 1789653953 · focus · HN ↗

      Having rolled my own RAG the other week, I personally would recommend talking through it with Claude Opus or a similar model.

      My baseline was (vibe coded) full-text search with SQLite, and we landed on long chunks with overlap, with a really simple nearest-neighbor search: quantize the index to a sign bit per scalar, which makes it super cheap to estimate dot products, then calculate better (8-bit quant index times native precision for the query) dot products to sort the top documents. A vector database would make sense for a much larger corpus, but I currently have fewer than two million rows. Claude vibe-coded it to use an OpenAI-speaking local inference server and made semantic search an optional augmentation for the full-text search.

      1. Neywiny · 1789699680 · focus · HN ↗

         I still don't understand this. Whenever I ask a model how to go this they recommend rolling my own. Is this really too niche to be readily available and well maintained in a GitHub but also so common every LLM knows countless ways to do it?

         1. entrope · 1789720933 · focus · HN ↗

            LLMs do tend to reinvent the wheel. There are off-the-shelf solutions. In-process, the major options seem to be Meta's Faiss, Spotify's Annoy and Alibaba's zvec. zvec's shared libraries are tens of megabytes in size, and I haven't looked closely at the others. My program is in Go, so even C bindings can be a pain.

            If you have billions of vectors, you should use a dedicated implementation: nearest-neighbor algorithms in high-dimensional space can be tricky, and there are a lot of trade-offs. My case is amenable to a simple implementation because I don't need huge scale. That's also why I did not need or want an out-of-process server.

Unofficial Hacker News client; not affiliated with Y Combinator.