Researchers asked frontier models to play chess. Have a look at the MAR rates in Table 3. When not explicitly told which moves were legal, no model identified legal moves at a rate better than 80%. Many asked for more illegal moves than legal moves. And even when explicitly told which moves were legal, the models continued to ask for illegal moves. With illegal asks discarded, none of the bots could beat a chess model calibrated to 1100 ELO.
The author of the originating post says that "current frontier models need laborious oversight and guardrails on even the simplest tasks", and he's absolutely correct.
This isn't how intelligence works. The LLM may not be able to play chess directly through inference, but it can write a program to do it and execute that program. Same as how human intelligence works. We can't fly, but we can build planes.
And how many years of direct play and study does it take for a human to get good at chess or any other game? Absolutely no human ever could be good at chess just by reading a few books, or even every book on chess. That's just not how the brain works. If LLMs could do that they would truly be superintelligence.
No, learning is definitely not a sign of super intelligence. I know words don’t mean anything anymore, but that is simply general intelligence, despite the claims we have reached this milestone.
We're obviously using different meanings for "good" here. But aside from that, it took 100's to 1000's of reinforcement iterations for your brother to play competently. While certainly impressive, that is still an entirely different category from piecing together disparate facts learned during training (LLMs aren't analyzing a board as they're learning the rules or ingesting PNG files), to executing a competent performance in one shot.
> Absolutely no human ever could be good at chess just by reading a few books, or even every book on chess.
Maybe not, but you'd be surprised how little it takes.
A six year old child can learn the rules of chess well enough to be able to play legal moves only in a single day. And they can improve their game at a pace which is almost frightening to behold. I have taught children, and I've witnessed significant improvement materialise in a single game. LLMs have probably thousands of chess books, games, videos, etc in their training data, yet they are unable to even follow the rules.
This is, at the very least, interesting. It illustrates many of the things brains can do, which current ML systems in general, and LLMs in particular, can't.
It is interesting, but people are drawing the wrong conclusion from it. For one, LLMs don't go through a "chess learning phase". They're not analyzing a board as they're learning the rules or studying games to create a coherent model of chess. They're just imbibing raw relationships as disparate fragments of information. The fact that they can't unify this into a coherent model of chess playing in one shot and execute a competent game of chess says nothing interesting about the limits of their intelligence. If you give frontier models the rules of chess in their context window, could they perform only legal moves? I bet they could, excepting trickier scenarios like pins and failing to respond to a check. But those kinds of scenarios have to be reinforced in any human player as well.
Poorly in what sense? I think human chess leagues are way more popular and fun than just playing a computer by yourself. Human oriented communities are always a vastly better experience than their digital counterparts.
There's more to games than simply winning you know.
carodgers · · focus · HN ↗
<a href="https://arxiv.org/html/2509.24239v4" rel="nofollow">https://arxiv.org/html/2509.24239v4
Researchers asked frontier models to play chess. Have a look at the MAR rates in Table 3. When not explicitly told which moves were legal, no model identified legal moves at a rate better than 80%. Many asked for more illegal moves than legal moves. And even when explicitly told which moves were legal, the models continued to ask for illegal moves. With illegal asks discarded, none of the bots could beat a chess model calibrated to 1100 ELO.
The author of the originating post says that "current frontier models need laborious oversight and guardrails on even the simplest tasks", and he's absolutely correct.
consensus1 · · focus · HN ↗
nefarious_ends · · focus · HN ↗
what · · focus · HN ↗
hackinthebochs · · focus · HN ↗
sph · · focus · HN ↗
hackinthebochs · · focus · HN ↗
lelanthran · · focus · HN ↗
Time is irrelevant to training; the more relevant comparison is "how many games does a human need to play to get diminishing returns".
hackinthebochs · · focus · HN ↗
orwin · · focus · HN ↗
hackinthebochs · · focus · HN ↗
mtlmtlmtlmtl · · focus · HN ↗
Maybe not, but you'd be surprised how little it takes.
A six year old child can learn the rules of chess well enough to be able to play legal moves only in a single day. And they can improve their game at a pace which is almost frightening to behold. I have taught children, and I've witnessed significant improvement materialise in a single game. LLMs have probably thousands of chess books, games, videos, etc in their training data, yet they are unable to even follow the rules.
This is, at the very least, interesting. It illustrates many of the things brains can do, which current ML systems in general, and LLMs in particular, can't.
hackinthebochs · · focus · HN ↗
thesmtsolver2 · · focus · HN ↗
consensus1 · · focus · HN ↗
shimman · · focus · HN ↗
There's more to games than simply winning you know.
thesmtsolver2 · · focus · HN ↗
qarl · · focus · HN ↗
But nobody wants that.
jstanley · · focus · HN ↗
Yes some people can do it but most people can't even if they're unusually intelligent.
You really need to be giving the LLM a board representation.
EDIT: I see that they actually were giving the LLMs a board representation and they still played badly. Fair enough then.