Researchers asked frontier models to play chess. Have a look at the MAR rates in Table 3. When not explicitly told which moves were legal, no model identified legal moves at a rate better than 80%. Many asked for more illegal moves than legal moves. And even when explicitly told which moves were legal, the models continued to ask for illegal moves. With illegal asks discarded, none of the bots could beat a chess model calibrated to 1100 ELO.
The author of the originating post says that "current frontier models need laborious oversight and guardrails on even the simplest tasks", and he's absolutely correct.
people on the ground know that small models are enough, since llms are good at directed work (i.e handholding) not the let loose go wild that the labs try to hype on.
the only thing that few people are willing to admit is that humans are the bottleneck as humans are needed to handhold / verify output - which puts a dent or might I say pause on the excessive valuations of a.i companies as that's against the narrative.
carodgers · · focus · HN ↗
<a href="https://arxiv.org/html/2509.24239v4" rel="nofollow">https://arxiv.org/html/2509.24239v4
Researchers asked frontier models to play chess. Have a look at the MAR rates in Table 3. When not explicitly told which moves were legal, no model identified legal moves at a rate better than 80%. Many asked for more illegal moves than legal moves. And even when explicitly told which moves were legal, the models continued to ask for illegal moves. With illegal asks discarded, none of the bots could beat a chess model calibrated to 1100 ELO.
The author of the originating post says that "current frontier models need laborious oversight and guardrails on even the simplest tasks", and he's absolutely correct.
dzonga · · focus · HN ↗
the only thing that few people are willing to admit is that humans are the bottleneck as humans are needed to handhold / verify output - which puts a dent or might I say pause on the excessive valuations of a.i companies as that's against the narrative.