‹ BackHN Continuity

Thread

HarnessTax: How Much Does the Harness Matter for Coding Agents?

233 points · 99 comments · matt_d

  1. jswelker · · focus · HN ↗
    Much of the extra weight of Claude Code and Codex harness are (ostensibly?) for security and alignment purposes. Whether they are effective is an open question, but leaving those dimensions out and calling it a tax is disingenuous, just turning insecurity into a negative externality.

    "Why pay the waste disposal tax? Dumping into the ocean is free!"

    Pi actively omits any sort of guardrails and sandboxing in the name of speed and simplicity, so it is not shocking that it is faster and simpler.

    Doubling the cost of something in the name of vague security is standard operating procedure for big enterprises, maybe even quite cheap.

    1. roywiggins · · focus · HN ↗
      It's not hard to sandbox Pi without adding anything substantial to the actual size of the system prompt, which is what HarnessTax is mostly measuring. Wrapping it in nono.sh costs approximately zero tokens.

      (I do think Pi should ship with more than zero builtin sandboxing though)

      1. imtringued · · focus · HN ↗
        The developers of pi.dev are geniuses.

        They add a --tools flag, which can only add tools onto the built-in tools, so if you wanted to sandbox pi.dev by adding sandboxed versions of the existing tools you will have to run pi.dev via

            pi --no-tools --tools tool1, tool2, tool3
        
        except..., that's not enough to sandbox pi.dev, because remember, it's a minimalist coding agent! So what does a minimalist coding agent do? Of course! It loads extensions by default and do you know what extensions can do? they can add default tools that bypass --no-tools!

        So if you want to sandbox your agent, guess what you'll have to do? Yep, you have to supply --no-extensions.

        Here is how to run pi.dev under its most minimal configuration under a sandbox:

            pi --no-tools --no-extensions --tools tool1, tool2, tool3 -e ./your-sandboxing-extension
        
        Pretty neat, huh?

        Sources: <a href="https:&#x2F;&#x2F;github.com&#x2F;earendil-works&#x2F;pi&#x2F;issues&#x2F;555" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;earendil-works&#x2F;pi&#x2F;issues&#x2F;555

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.