Thread

1. linux_devil · 1785992531 · focus · HN ↗

   Why do we need to train the model to solve for retrieval within the org, so we have to keep training it whenever new dataset is introduced , or am I missing something here ?

   1. jmalicki · 1785993265 · focus · HN ↗

      It's a matter of cost. Did you see the 100x cheaper?

      If you have a workload that is going to be very heavy, incurring a large training cost to make a cheaper model work well with the dataset will be dramatic cost reduction. Most large AI workloads can't afford, or truly need, the expense or capability of GPT 5.6 Sol when cheaper models can do.

      Of course you could skip that and just use GPT-5.6 Sol everywhere instead. If you're running a fast food restaurant you could hire Michelin star chefs to make your burger and fries without further training. Or you could have a training program for teenagers, a sourcing program, etc. to scale up to your chain to still get consistent quality without needing that level of cost in each store, but replacing it with a centralized repeatable process.

   2. kumama · 1785994302 · focus · HN ↗

      (founder of castform here) the model you post-train should ideally learn general patterns & search strategies over your dataset that should transfer to new docs you add to the search corpus (unless its super out of distribution)

      1. srvraw · 1786006827 · focus · HN ↗

         could you share more about what you mean by "general patterns & search strategies"? I can think of it being along the lines of searching over specific tables or databases for queries in certain context. It's an exciting line of work and I'm interested because I need something like this for the problem I'm solving atm. So, I'd like to understand how the training generalizes

         1. kumama · 1786027497 · focus · HN ↗

            yup! it’s mostly about getting better at using the right search keywords.

            for more complex multi-hop question, it's also about knowing which sections of a document to look up and in what order.

      2. oh_no · 1786021937 · focus · HN ↗

         i'm a little confused by this, should most piles of company documentation look pretty similar? what are we getting by tuning at the org level?

         what if you have a bunch of teams or apps that have different documentation patterns?

         how much does this degrade over time, it beats leading models with that static data set but clearly this edge will degrade with data drift, how quickly does that happen?

         also, any "this is 100x cheaper" blogpost means nothing if not discussing TCO (I know your team didn't write this.) I don't care what inference costs are if I don't know training/overhead costs. what's the breakeven point. and again, how long is this RAG stack going to be worth keeping, you beat 5.6 Luna but at some point un-tuned models will beat you, so this is a temporary solution that needs to be re-upped at some point. benchmarks against data drift would help there

         1. kumama · 1786027103 · focus · HN ↗

            very useful questions here!

            - if you have teams that have totally different documentation patterns, yes i think a model trained one teams' docs probably won't directly transfer to another. but let's say you have a model trained on SEC 10-K filings, it should work well as new 10-K filings are added (since terminology & structure of these docs are similar) - on TCO, the training costs for the experiments you saw were <$200 - you bring up a good point on data drift, don't have concrete numbers/experiments here. something we should defn look into (though i imagine this also changes a lot based on use-case and company)

Unofficial Hacker News client; not affiliated with Y Combinator.