I am increasingly hesitant to use non-native harnesses - model providers are now starting to train their agents for use within the harness. An eval like terminal bench can only capture so much data. I don't want to have to assess each harness every model release to make sure it's working as well as it can.
Right! I had the exact opposite view when reading this comment. The good timeline is where one of (or perhaps a small number of) the open source harnesses becomes so dominant that the models compete to have the model that is the best trained to work with that harness. (I'm hoping this would be Pi, because it's my favorite, but mostly I just want it to be some model-agnostic open source harness that wins.)
Realistically, it's never going to be Pi, for the same reason that video game consoles are always more performant than PCs for a given hardware spec.
There's no canonical Pi harness, it's too variable. There might be a world where something Pi based is the target, like OMP, but then you have to hesitate when you start extending the harness because you don't know what effect any given extension will have on model performance.
johnmlussier · · focus · HN ↗
stogot · · focus · HN ↗
Cost efficiency is a plus
sanderjd · · focus · HN ↗
Silagi · · focus · HN ↗
There's no canonical Pi harness, it's too variable. There might be a world where something Pi based is the target, like OMP, but then you have to hesitate when you start extending the harness because you don't know what effect any given extension will have on model performance.
sanderjd · · focus · HN ↗