‹ BackHN Continuity

Thread

HarnessTax: How Much Does the Harness Matter for Coding Agents?

233 points · 99 comments · matt_d

  1. nojs · · focus · HN ↗
    We really need better harness benchmarks. It seems there's no reliable source that benchmarks the main harnesses against all open source models.

    I also wish the discussion around Pi did not always use cost/token count as the metric. It's amazingly token efficient, but how does it stack up again opencode and others if you don't care about token count?

    My experience is that the harness is mainly polish preventing failed tool calls, bad edits, stuff like that, but doesn't make much difference to the overall "intelligence". But that opencode seems slightly more robust against stupid errors than out of the box Pi due to the additional context it forces through every thread.

    1. sn0n · · focus · HN ↗
      A good benchmark would require a decent number of smaller scoped one off tasks to larger multi step refactors, and also one shot full project of simple to complex varieties. In addition to a series of “conversational” ambiguity filled one-liners.
    2. leemysw · · focus · HN ↗

      [dead]

    3. calgoo · · focus · HN ↗
      I really disliked how opencode works IMO; the harness tries to do to much in my mind. Switching to Pi was a breath of fresh air for me, and I even use hax for some of my local needs where i dont want to have the giant pile of fertilizer that is NPM or PIP installed.

      The harness becomes more and more important, the smaller the model is as you need to offload context management as well as memory to the harness. The big models basically just need a bash prompt tooling and you let the model manage everything inside its own context.

      1. mobelkh · · focus · HN ↗
        what does it do too much out if the box? afaik it only has the basic tools and two agents profiles, maybe lsp support is extra?
        1. tacomagick · · focus · HN ↗
          Moving from CC I can agree OpenCode seems very simple. I don't see what is too much about it honestly.
        2. verdverm · · focus · HN ↗
          look into the prompts, there are some choices in there

          disclaimer, I use opencode and have customized parts of it, and will do more, but it is a solid foundation and comes with more out of the box than pi

          pi is too minimal for me, I'd go back to my custom built harness if I wanted to be back at that level

          1. esperent · · focus · HN ↗
            > look into the prompts, there are some choices in there

            Such as?

            1. verdverm · · focus · HN ↗
              Instructing to use the `gh` CLI, not ideal for those of use using BBC at work

              They also have instructions about how to format certain output, which conflicts with the instructions we have in repo. I only discovered yesterday because we were wondering why the agent kept picking certain tools.

              1. esperent · · focus · HN ↗
                What is BBC? Other than the tv network.
                1. verdverm · · focus · HN ↗
                  Bitbucket Cloud (Atlassian)
                2. [deleted] · · focus · HN ↗

                  [deleted]

    4. sandeepkd · · focus · HN ↗
      Harness and benchmark for the harness feels like a chicken and egg problem. The harness is to optimize the interaction results with the models. Any benchmark for harness has to focus on the goals that the harness was trying to optimize for unless we are only focussing on generic harnesses.

      At this point when all the models have been trained on all available data with the similar algorithm,

      1. either you get more data which is not feasible,

      2. or get a better algorithm - a possibility ,

      3. or write a more targeted harness.

      Harnesses for legal, medicine and all are the ones which are getting focus for this reason. Writing benchmarks for these targeted harnesses would be a catching task

    5. general_reveal · · focus · HN ↗

      [dead]

      1. tomhow · · focus · HN ↗
        > lizard satanists

        > Tell a horny monkey not to jerk off.

        Can you just not post garbage like this on HN. It's okay to criticize the big tech companies or general AI discussion here. Many do, it’s fine. But dreck like that only makes you seem unhinged and is the surest way to turn this place into the cesspool you say you're concerned about. Honestly. Some people have to read this stuff whether they want to or not. And to post this utter filth in a comment that's appealing for higher standards? Good grief.

        <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;newsguidelines.html">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;newsguidelines.html

    6. ebrahimisoheil · · focus · HN ↗

      [dead]

    7. tomrod · · focus · HN ↗
      I&#x27;ve been thinking about this a lot in the past few weeks - <a href="https:&#x2F;&#x2F;www.roderick.dev&#x2F;writing&#x2F;2026-08-28-obsessing-harnesses&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.roderick.dev&#x2F;writing&#x2F;2026-08-28-obsessing-harnes...

      I don&#x27;t think yet that a general benchmark will capture what is needed, but a gain&#x2F;loss of function along with improvement &#x2F; loss along the ability of that function seems to be a good path forward.

    8. artdigital · · focus · HN ↗
      Very curious as well to see comparisons between Claude Code, Codex, Pi, Grok, OpenCode, Crush, etc

      Personally jumping around a lot to get a feeling for exactly that, and these days liking the Grok harness out of all of them the most

    9. verdverm · · focus · HN ↗
      Coming Soon: <a href="https:&#x2F;&#x2F;artificialanalysis.ai&#x2F;agents&#x2F;coding-agents?coding-agents-performance-chart=swe-atlas-qna&amp;coding-agents-harness-comparison-chart=harness-deep-swe#harness-comparison" rel="nofollow">https:&#x2F;&#x2F;artificialanalysis.ai&#x2F;agents&#x2F;coding-agents?coding-ag...

      They used to have one chart with {model X harness} for a subset of combos, looks like that is getting an upgrade

    10. jiaosdjf · · focus · HN ↗
      We really do need better benchmarks and for models too

      - Most people use a harness because of its subscription (most companies pay Anthropic) - All model benchmarks are biased and gamed, harness benchmarks are too few to matter - Everyone is just guessing, acting on sample sizes of 1 and trust me bro vibes

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.