‹ BackHN Continuity

Thread

Fable 5 – Median thinking declined in August

428 points · 293 comments · espeed

  1. cloudking · · focus · HN ↗
    How do you create repeatable tests in a non-deterministic system? Every time you send the same prompt you get a different answer.
    1. ssivark · · focus · HN ↗
      The actual tokens might be non-deterministic, but you could look for proxy measures that are supposed to be invariant. Eg. correctness/performance on benchmarks, "thinking level" on complex problems, etc
    2. CharlesW · · focus · HN ↗
      This is a good overview of how this is done: <a href="https:&#x2F;&#x2F;www.anthropic.com&#x2F;engineering&#x2F;demystifying-evals-for-ai-agents" rel="nofollow">https:&#x2F;&#x2F;www.anthropic.com&#x2F;engineering&#x2F;demystifying-evals-for...
    3. 6gvONxR4sf7o · · focus · HN ↗
      That&#x27;s like the entire field of statistics.
Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.