Researchers asked frontier models to play chess. Have a look at the MAR rates in Table 3. When not explicitly told which moves were legal, no model identified legal moves at a rate better than 80%. Many asked for more illegal moves than legal moves. And even when explicitly told which moves were legal, the models continued to ask for illegal moves. With illegal asks discarded, none of the bots could beat a chess model calibrated to 1100 ELO.
The author of the originating post says that "current frontier models need laborious oversight and guardrails on even the simplest tasks", and he's absolutely correct.
It is even worse.. This is a classical reinforcement problem where data generation is easy because the rule set is pre-defined. So you really don't even need any data to start with (but would help).
There are more possible game combinations than atoms in the universe, even those generation of valid game states are as you say pre-defined. that is why models cannot go this route and therefore are poor at chess
Isn’t this exactly how AlphaZero was trained? The rules are known and well defined so the training process can generate games without any outside data.
The only reason LLMs are this bad at chess is because the labs don’t care about chess performance so they’re not going out of their way to train the models for it. The ability they do have is from what chess information happens to be in the training data, plus whatever general reasoning abilities they may be able to apply.
I think the line of criticism around LLMs sucking at chess makes more sense when you understand what the AI companies are saying about the future trajectory of these models.
The entire recursive self improvement story falls apart once you point out that there is not much "cross domain transfer learning". Meaning that training an LLM to become good at coding, math, etc, will eventually transfer into them being good at other skills that were not explicitly trained for.
Using games like chess which have little economic value is actually a good test for this. What's even more surprising about them sucking at chess is how much information about chess strategy exists in the training data.
There’s literature. But I don’t think there’s reams of chess games. With literature the LLM can understand strategy but chess needs intuition and you need tokenized games for that. LLMs have less of that.
There’s multiple databases of games in algebraic notation. You can also, very easily rl train on pitting models against one another, even without mcts.
there are far more reams of chess moves than there are math papers. Lichess is pretty open...
But hey, they're actually good at chess if you prompt correctly so....
<a href="https://dynomight.net/more-chess/" rel="nofollow">https://dynomight.net/more-chess/
carodgers · · focus · HN ↗
<a href="https://arxiv.org/html/2509.24239v4" rel="nofollow">https://arxiv.org/html/2509.24239v4
Researchers asked frontier models to play chess. Have a look at the MAR rates in Table 3. When not explicitly told which moves were legal, no model identified legal moves at a rate better than 80%. Many asked for more illegal moves than legal moves. And even when explicitly told which moves were legal, the models continued to ask for illegal moves. With illegal asks discarded, none of the bots could beat a chess model calibrated to 1100 ELO.
The author of the originating post says that "current frontier models need laborious oversight and guardrails on even the simplest tasks", and he's absolutely correct.
threethirtytwo · · focus · HN ↗
The caveat is: It depends on the task.
Are there reams of chess moves that the model can train off of? No.
Are there reams of math papers the model can train off of? Yes.
keephnacct · · focus · HN ↗
[dead]
[deleted] · · focus · HN ↗
[deleted]
iwontberude · · focus · HN ↗
[dead]
tjwebbnorfolk · · focus · HN ↗
This is as false as something can possibly be. There are open databases of millions of chess games spanning hundreds of years.
XenophileJKO · · focus · HN ↗
manquer · · focus · HN ↗
wat10000 · · focus · HN ↗
The only reason LLMs are this bad at chess is because the labs don’t care about chess performance so they’re not going out of their way to train the models for it. The ability they do have is from what chess information happens to be in the training data, plus whatever general reasoning abilities they may be able to apply.
threethirtytwo · · focus · HN ↗
Was there reams of chess moves that the model trained off of? No.
vmg12 · · focus · HN ↗
I think the line of criticism around LLMs sucking at chess makes more sense when you understand what the AI companies are saying about the future trajectory of these models.
The entire recursive self improvement story falls apart once you point out that there is not much "cross domain transfer learning". Meaning that training an LLM to become good at coding, math, etc, will eventually transfer into them being good at other skills that were not explicitly trained for.
Using games like chess which have little economic value is actually a good test for this. What's even more surprising about them sucking at chess is how much information about chess strategy exists in the training data.
freejazz · · focus · HN ↗
For real??
threethirtytwo · · focus · HN ↗
freejazz · · focus · HN ↗
threethirtytwo · · focus · HN ↗
FuckButtons · · focus · HN ↗
thelaxiankey · · focus · HN ↗
But hey, they're actually good at chess if you prompt correctly so.... <a href="https://dynomight.net/more-chess/" rel="nofollow">https://dynomight.net/more-chess/