Thread

Beating GPT-5.6 Sol on retrieval with 100x cheaper open models

414 points · 114 comments · moonikakiss

nullbio · 1785998912 · focus · HN ↗
I (and I imagine many others) would love to use something like this, but can't, because my data is too sensitive to be uploaded to a cloud of which I have no gaurantees of privacy/security.
Is there any way we do this using rented GPUs and open-source software stacks? Paying for the service isn't the issue, I don't care if it's free or if a cut is taken in some capacity, I just don't want the provider to have access to my data.
1. i5heu · 1786000576 · focus · HN ↗
  In OpenRouter there are Zero Retention options. And if you use a EU provider you can be somewhat sure that your privacy is given.
  Other than that there is not really a difference to renting a GPU since the GPU provider can also just steal your data.
  Local GPU(s) are always an option if you have the possibility. It is also not that difficult to run with stuff like “LocalAI”
2. kumama · 1786027808 · focus · HN ↗
  castform founder here. unfortunately, we are cloud-hosted at this point.
  but some easy options on the open-source side include huggingface's trl & unsloth.
  you can run our data-generation scripts here: <a href="https://github.com/castform-ai/benchmax" rel="nofollow">https://github.com/castform-ai/benchmax and then hook it up trl/unsloth for training. should be able to do all of this on your own compute.

Unofficial Hacker News client; not affiliated with Y Combinator.