This is done on Nemotron models + mistral, so its not very relevant to the current frontier of cheap chinese models + big models from Claude/GPT. Big miss not having qwen or deepseek in this research.
The focus of the study was the different harness approaches and how they scale across model sizes. The fact that they used any particular set of models is irrelevant.
I'm not totally convinced that models are fungible, the claudes/gpts/Gemini all have pretty individual feels when you're working with them. I wouldn't be surprised if the approaches don't scale or even work the same in a poly model setup
that's anecdotal though, right? your subjective feeling of how a model responds to you will greatly influence how you 'feel' about a model and its performance in the same way that a co-worker who you get along with will fuck something up and you'll be more forgiving than when you work with a too-verbose, mansplainer of a co-worker who fucks up
I think until we have actual repeated-use measurements tracked over time (eg consistent prompts used to do the same tasks, count number of hallucinations and errors and bugs over a long period of time) you won't really have any idea of which model is better
I also think of it like a car - some just feel better to drive even if they are materially worse in other measures. until you start measuring the metrics important to you (eg MPG and cost of maintenance over a long period), you have no idea which car is actually better suited for you. and the fact that you can only do so with a limited number of cars (or hours available to work, or money to burn on tokens) means there's no true measure approaching objectivity
You can write a config that points your claude code at any anthropic-compatible endpoint, and replace opus and sonnet with whatever you want. Things like deepseek and glm in there do not feel at all like they do in more minimal harnesses, but neither do they convincingly act like claude models. It's very stark and frankly they run like shit this way. Try it.
vblanco · · focus · HN ↗
dsiegel2275 · · focus · HN ↗
klooney · · focus · HN ↗
paimapi · · focus · HN ↗
I think until we have actual repeated-use measurements tracked over time (eg consistent prompts used to do the same tasks, count number of hallucinations and errors and bugs over a long period of time) you won't really have any idea of which model is better
I also think of it like a car - some just feel better to drive even if they are materially worse in other measures. until you start measuring the metrics important to you (eg MPG and cost of maintenance over a long period), you have no idea which car is actually better suited for you. and the fact that you can only do so with a limited number of cars (or hours available to work, or money to burn on tokens) means there's no true measure approaching objectivity
dcss_gardener · · focus · HN ↗
paimapi · · focus · HN ↗