Thread

Brood War Bench

346 points · 156 comments · benswerd

GodelNumbering · 1789850207 · focus · HN ↗
A friend of mine created GoBench[1][2] that evaluates LLMs on 9×9 Go using KataGo opponents as Elo anchors, you see real capability differences there, like Astra Max substantially leading all other models. I think strategy is a generally interesting area to evaluate LLMs on
[1] <a href="https://rolandgao.com/blog/gobench/" rel="nofollow">https://rolandgao.com/blog/gobench/
[2] <a href="https://rolandgao.com/gobench.pdf" rel="nofollow">https://rolandgao.com/gobench.pdf
1. monster_truck · 1789882717 · focus · HN ↗
  imho doing it on 9x9s kind of misses the whole point, anyone who has gotten to single digit kyu vs humans will understand when I say its effectively tic tac toe with extra steps
  1. Shorel · 1789891750 · focus · HN ↗
    It's a very fun tic tac toe!
    (I have not reached single digit kyu)
    monster_truck · 1789898201 · focus · HN ↗
    Don't let yourself get too comfy with 9x9! I stayed on it for too long. No way to know but I feel like it added 2 years to the time it took to get there

Unofficial Hacker News client; not affiliated with Y Combinator.