Thread

1. Supermancho · 1789605812 · focus · HN ↗

   The term "harness" here is being overloaded for the term "agent", which is worrying. Putting that aside, there are many factors that matter. The "harness" context, the execution pattern (parallel vs sequential), the ability to delegate to other models, etc.

   Optimal harnesses use concurrent execution + subagents and are not stuck on one model. Cost and performance are impacted GREATLY by these tactics, regardless of the native agent context (instruction). This kind of single-harness analysis is shallow and misleading, although the finding that "Provider-specific optimization does not guarantee the best pairing" is probably correct, depending on how you measure.

   It is a starting point.

   1. corv · 1789621768 · focus · HN ↗

      While sub-agents could theoretically stack turtles indefinitely, there’s limited room when going up the stack because someone will have to take responsibility beyond “git blame”, and that lands squarely with a person or legal entity.

      1. visarga · 1789636387 · focus · HN ↗

         A big part of that responsibility can be put in code tests.

         Ensuring good test coverage and quality is how you purchase trust in the work agents do. This also reduces the context problem - a test collection has no recall issues, it just runs every time you call it, the whole battery, checks all the things we could check by code in one fast tool call. For the rest, the things we can't test by code, I use manual testing.

         A large class of problems are intent divergence, when the model passes tests but it didn't do what I asked. For that I keep a log of all user messages in the project history and review it with agents. This intent alignment is repeated from time to time to catch drift.

         So I see the "why should I trust the work agent did?" problem as a combination of 1. ensure good testing 2. review intent alignment.

   2. togilvie · 1790167443 · focus · HN ↗

      Starting point is the right framing. Feels like every harness will need tools to identify the best model for the task, best tools, etc. Ultimately, the only metric that matters is cost-per-successful-task, and it's good we're starting to measure these things. But need a lot more discipline on it.

Unofficial Hacker News client; not affiliated with Y Combinator.