Thread

1. epolanski · 1789647104 · focus · HN ↗

   Okay but...closed source harnesses change multiple times per week, sometimes per day.

   And they also tune the inference of the models behind it.

   You're using different tooling every day. Hard to benchmark.

   1. edgyquant · 1789648467 · focus · HN ↗

      A benchmark would be a series of prompts + tests that check their results you theoretically run every day

Unofficial Hacker News client; not affiliated with Y Combinator.