This is done on Nemotron models + mistral, so its not very relevant to the current frontier of cheap chinese models + big models from Claude/GPT. Big miss not having qwen or deepseek in this research.
The focus of the study was the different harness approaches and how they scale across model sizes. The fact that they used any particular set of models is irrelevant.
I'm going to cherry pick one example where newer models are noticeably improving at least in my experience.
What is a noticeable improvement with something that struggles to read a message longer than 200 characters without missing information in the middle, may be a 0.000000001% improvement with a model that... almost never misses info in the first place.
vblanco · · focus · HN ↗
dsiegel2275 · · focus · HN ↗
hiddencost · · focus · HN ↗
bjelkeman-again · · focus · HN ↗
belowavgiq · · focus · HN ↗
What is a noticeable improvement with something that struggles to read a message longer than 200 characters without missing information in the middle, may be a 0.000000001% improvement with a model that... almost never misses info in the first place.