Thread

1. jeremyjh · 1790449271 · focus · HN ↗

   You only need to spend a couple of minutes talking to an LLM about chess to realize it has no understanding of the game. It will produce reasonable sounding explanations for things that are completely wrong. It will make illegal moves. It will hallucinate pieces on squares even when the entire PGN is in the context. There are no prompts you can give it to fix this. I don't really understand how you could create this skill without noticing this.

   1. Lucasoato · 1790449555 · focus · HN ↗

      Will it make illegal moves? I have doubts regarding this, at least for the strongest models like Opus5.5 or Astra.

      1. jeremyjh · 1790449869 · focus · HN ↗

         Absolutely it will. You have to run them in an environment without tools because today they just turn on stockfish if you ask on claude.ai.

         These models are strong at math and coding because they have been heavily trained for it in post-training. That is where a majority of the gains have come from in the past two years. That has absolutely no impact on chess ability. Heavily train a deep learning model for chess and you get Leela, an absolutely monster. Don't train them for chess and I doubt its even 1200.

         It is possible to do post-training on a base LLM and get a strength equivalent to a strong amateur but people haven't done that at any meaningful scale recently, so hard to say what the ceiling might be with the largest models. Probably pretty high but it would be expensive and still worse than Leela.

         The only interesting question is could it explain chess to you - and I haven't seen evidence that it can. ChessGPT at least - while purportedly 1800 strength, can't explain strategic themes from the first chapter of beginner strategy books accurately or explain why specific moves are bad (positionally), it just knows they are.

         <a href="https:&#x2F;&#x2F;github.com&#x2F;waterhorse1&#x2F;ChessGPT" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;waterhorse1&#x2F;ChessGPT

         1. brumar · 1790451678 · focus · HN ↗

            Yep, they are quite bad without stockfish. You can test it with the playchess skill in my repo. Maybe 1200&#x2F;1300, who knows? Still I do think that they can, with enough time, explore multiple variations where they confront their naivety to stockfish and build up a compact picture on why move Y should have been played instead of move X. That was my intuition when building this skill.

            1. jeremyjh · 1790453481 · focus · HN ↗

               Have you actually tested this with quiet positions, where understanding needs to be more about different strategic or positional choices? They can explain concrete tactical lines with stockfish analysis at hand, but so can an analysis board. In my experience, even with fairly deep and accurate engine analysis, they cannot do this accurately with quiet positions. I tried to build a product around this idea and I&#x27;m convinced that even if it is possible it would require hundreds of thousands - or millions - spent on post-training to even confirm that possibility.

               1. brumar · 1790455273 · focus · HN ↗

                  Good point. I had to tweak a bit the system to get more positional analysis because this is something I wanted too get. I think it&#x27;s still shying away on this aspect. When it does dwell on it, I feel that when the output describe what should be the plan of both camps, it&#x27;s quite convincing but only a much stronger player (or me with stockfish) could really assess this.

                  Anyway I am with you that deep positional appraisal is very hard and we should not expect too much from this set of skills on that front.

                  1. [deleted] · 1790456780 · focus · HN ↗

                     [deleted]

Unofficial Hacker News client; not affiliated with Y Combinator.