Researchers asked frontier models to play chess. Have a look at the MAR rates in Table 3. When not explicitly told which moves were legal, no model identified legal moves at a rate better than 80%. Many asked for more illegal moves than legal moves. And even when explicitly told which moves were legal, the models continued to ask for illegal moves. With illegal asks discarded, none of the bots could beat a chess model calibrated to 1100 ELO.
The author of the originating post says that "current frontier models need laborious oversight and guardrails on even the simplest tasks", and he's absolutely correct.
there are far more reams of chess moves than there are math papers. Lichess is pretty open...
But hey, they're actually good at chess if you prompt correctly so....
<a href="https://dynomight.net/more-chess/" rel="nofollow">https://dynomight.net/more-chess/
carodgers · · focus · HN ↗
<a href="https://arxiv.org/html/2509.24239v4" rel="nofollow">https://arxiv.org/html/2509.24239v4
Researchers asked frontier models to play chess. Have a look at the MAR rates in Table 3. When not explicitly told which moves were legal, no model identified legal moves at a rate better than 80%. Many asked for more illegal moves than legal moves. And even when explicitly told which moves were legal, the models continued to ask for illegal moves. With illegal asks discarded, none of the bots could beat a chess model calibrated to 1100 ELO.
The author of the originating post says that "current frontier models need laborious oversight and guardrails on even the simplest tasks", and he's absolutely correct.
threethirtytwo · · focus · HN ↗
The caveat is: It depends on the task.
Are there reams of chess moves that the model can train off of? No.
Are there reams of math papers the model can train off of? Yes.
thelaxiankey · · focus · HN ↗
But hey, they're actually good at chess if you prompt correctly so.... <a href="https://dynomight.net/more-chess/" rel="nofollow">https://dynomight.net/more-chess/