Thread

Show HN: Raven – The harness of harnesses, built for RSI

55 points · 51 comments · cyfyifanchen

redhale · 1790701326 · focus · HN ↗
I don't want to be too negative, but ... all this for a 0.8% improvement in SWE-bench Verified (90.2 for OpenCode vs 91)? And why is this (saturated) benchmark the one coding benchmark chosen to showcase on the homepage?
Without trying it, this seems like its probably just a massive waste of tokens.
1. PcChip · 1790705790 · focus · HN ↗
  this might be a dumb question, but if it's saturated, then isn't 0.8% improvement a very good improvement?
  1. redhale · 1790710528 · focus · HN ↗
    I'm definitely willing to be corrected, but I think it just means that this improvement is close to pure noise -- not good or bad necessarily. Just not a good signal.
    To me the really bad sign is that this is the benchmark that they selected to highlight. Why not one of the less-saturated benchmarks where this harness could (in theory) show meaningful improvement over the OpenCode baseline? Seems fishy to me.

Unofficial Hacker News client; not affiliated with Y Combinator.