Thread

1. nojs · 1789618135 · focus · HN ↗

   We really need better harness benchmarks. It seems there's no reliable source that benchmarks the main harnesses against all open source models.

   I also wish the discussion around Pi did not always use cost/token count as the metric. It's amazingly token efficient, but how does it stack up again opencode and others if you don't care about token count?

   My experience is that the harness is mainly polish preventing failed tool calls, bad edits, stuff like that, but doesn't make much difference to the overall "intelligence". But that opencode seems slightly more robust against stupid errors than out of the box Pi due to the additional context it forces through every thread.

   1. calgoo · 1789626263 · focus · HN ↗

      I really disliked how opencode works IMO; the harness tries to do to much in my mind. Switching to Pi was a breath of fresh air for me, and I even use hax for some of my local needs where i dont want to have the giant pile of fertilizer that is NPM or PIP installed.

      The harness becomes more and more important, the smaller the model is as you need to offload context management as well as memory to the harness. The big models basically just need a bash prompt tooling and you let the model manage everything inside its own context.

      1. mobelkh · 1789636304 · focus · HN ↗

         what does it do too much out if the box? afaik it only has the basic tools and two agents profiles, maybe lsp support is extra?

         1. verdverm · 1789655755 · focus · HN ↗

            look into the prompts, there are some choices in there

            disclaimer, I use opencode and have customized parts of it, and will do more, but it is a solid foundation and comes with more out of the box than pi

            pi is too minimal for me, I'd go back to my custom built harness if I wanted to be back at that level

            1. esperent · 1789657513 · focus · HN ↗

               > look into the prompts, there are some choices in there

               Such as?

               1. verdverm · 1789658448 · focus · HN ↗

                  Instructing to use the `gh` CLI, not ideal for those of use using BBC at work

                  They also have instructions about how to format certain output, which conflicts with the instructions we have in repo. I only discovered yesterday because we were wondering why the agent kept picking certain tools.

                  1. esperent · 1789662557 · focus · HN ↗

                     What is BBC? Other than the tv network.

                     1. verdverm · 1789664138 · focus · HN ↗

                        Bitbucket Cloud (Atlassian)

Unofficial Hacker News client; not affiliated with Y Combinator.