Researchers asked frontier models to play chess. Have a look at the MAR rates in Table 3. When not explicitly told which moves were legal, no model identified legal moves at a rate better than 80%. Many asked for more illegal moves than legal moves. And even when explicitly told which moves were legal, the models continued to ask for illegal moves. With illegal asks discarded, none of the bots could beat a chess model calibrated to 1100 ELO.
The author of the originating post says that "current frontier models need laborious oversight and guardrails on even the simplest tasks", and he's absolutely correct.
> Gemini 2.5 Pro, O3, Claude Sonnet 3.7 and ChatGPT 4.1
The gap in capabilities between those models which they tested, and actual current frontier ones is enormous. I would not trust that any conclusions they made are applicable.
It's also worth noting that the very latest models (GPT-6 and Fable 5.1) actually play worse than their immediate predecessors, so it is likely that the labs are not benchmaxxing for this yet. If they did, I'm sure they could come up with something superior to humans. But there is probably very little demand for this compared to IT stuff.
Even if you take that website at face value, the ELO scores shown are relative to the other AI models tested, and not comparable to the ELO scores of humans who play against other humans.
I wonder why they didn’t throw a real chess engine in there for a baseline. There are engines where you can set the elo in the settings, so it should possible to see these LLMs relative to a human 1500 rather than just relative to each other.
This is true, but I'm not sure it matters? I was poking around at the lichess database recently and those elo calibrated bots are remarkably well calibrated, their rating variance sticks out like a sore thumb compared to human players even at similar game volumes. So it should still be a decent predictor of how good a human at that level is, even if the playstyle seems alien.
This is simply blatant misinformation. If you play a game online on lichess and go to the analysis board you can find when your game becomes novel. It will be within 20 turns unless you are intentionally following a known opening. In fact it will likely become unique within 10-15 turns.
I am around 1500 (actually 1649 on lichess blitz, but close enough).
I explored the last 5 games I played on lichess. Here are the number of moves before lichess had never seen that position before for each of the 5 games: 6, 12, 16, 15, 11.
> If you find yourself in a novel position within 10-15 moves it's likely a resignable one
The opponent is also going to be in a novel position. Should both players resign?
Just in case you think this is limited to low rated players like 1500s, look at MagnusCarlsen's most used account on Blitz: <a href="https://lichess.org/@/DrNykterstein/search?perf=2" rel="nofollow">https://lichess.org/@/DrNykterstein/search?perf=2
You will see that most games become unique to the whole of lichess within 15-20 moves and a good chunk between 10-15.
Probably tells us that without labs explicitly training/tuning the models or designing the harness (with fast oracle) the LLMs aren't going to get good at those areas.
I know HN readers and posters just read numbers and can't be bothered to read, but please read the methodology before making any claims.
> About their ELO ratings from their own website:
> A field-relative rating calculated within ChessBench. It compares performance among the tested models and is not a direct equivalent of a human chess rating.
I am around 1600 elo in over the board I can mop up Astra Fable etc even if I give them literal infinite time and all the subagents and internet access..
Please folks at least use your AIs to read stuff before making claims.
AI is not GM level, it's not even 1600, I am 1600 by using memorized openings people frequently fall for with very basic intuitions.
A GM is 2600 they can beat me in under 20 moves...
Why do I even scroll through this website. For a moment I truly felt fooled, but then I read like a human should.
Maybe I should stop doing that will be a happier life, don't think just believe in the AGI.
Sub 1300 that's my rating in the singular official tournament I participated at.
But given how easily I can crush them and how often they want to make illegal moves (btw above bench seems to use a harness that pokea the model until it gives valid moves).
I would rate them around 500-800 big range but at that level it's all about if the model can recall an opening or not. If it plays good first 4-8 moves the person on the end will fumble for certain and they win.
I can play good/best moves till 14-15 moves if I remember the lines and find someone who falls for it.
If you could give them the lines as prompts like the best 20-30 openings then they will be around 700-800.
700 is around the rating for a human who doesn't know the tricks but can do bare minimum calculations and understands the rules thoroughly.
As someone who used to compete for years and plays currently as a hobbyist, you’re absolutely correct. LLM’s are terrible at chess and if anyone wants to sober up their view on AI, try it yourself.
Anyone who casually plays on a regular basis can beat them more often than they lose. As you said if you just know the core openings (and end games, both of which you can get a handle on with modest effort) you will generally win.
Edit: reminder we had computers beating the best players in the world literally decades ago. LLM’s are remarkable tools but the current promises and expectations are ridiculous
You can take LLMs out of opening knowledge by playing chess960, and their performance degrades significantly. I just tried playing Claude Sonnet 5 (high), and it made its first illegal move on move 5.
They played 4...c6, followed by 5...Nc6, somehow forgetting about the pawn the just put on c6. (My move in between was 5. Nc3, and apparently they were trying to mirror me.)
So you can see an actual game on that website, and the play seems pretty decent to me for a while (~1700 lichess = 1300 elo) until move 28 when black throws away their queen for absolutely no reason in an incomprehensible blunder.
In some ways this is reflective of the AI experience at large, sometimes shockingly competent but then also sometimes ludicrously incompetent.
The AI can write a chess bot program that will beat you.
You're thinking about this the wrong way. The system is built and delivered as it is because that's how the providers make the most money. If they cared to have it perform well in chess games, you'd see a different shape and behavior.
We shouldn't ask the multibillion dollar automated software generation system to play games with us any more than we should ask a Boeing's flight guidance system to do so.
I'm stating that certain folks are trying to use the software-generating product as an AGI/ASI and then complaining when it doesn't play chess very well.
People are holding it wrong, deliberately or not. Some are inventing bad faith measures so they can claim AI sucks.
Then why respond at all for the sake of responding?
We all know AI can code, but the question it all stemmed from what if it's AGI or GM level in chess on it's own.
You can't just back pedal from the statement that apparently being able to code a chess engine is the same as being good at chess.
I can write a chess engine that beats Magnus Carlson without AI that alone neither makes me GM level or AGI or any of the other claims the above comments seem to be making?
I agree w/ this perspective. An agent with a harness that can run programs can solve a lot more than one without the harness. The AI system includes the harness, and it's not clear to me that AGI requires more than LLMs + code generation & execution are capable of.
So AI is AGI in fields where code can't solve anything?
Is code omnipotent, I have been in software all my life and I would hard agree here.
Sure stuff LLMs can do with being good at parts of code reproduction is incredible. And honestly it's the new way to do a lot of things but I have not see an iota of proof that it can scale across the board.
For instance Maths is just code with different symbols and slightly less universally legible concepts.
AI is the best invention at figuring out or walking the search space and directionally doing logically computation over general software adjacent stuff.
But that's it, I am certain a bunch of companies will make a lot of money despite no AGI.
I think people either don't understand AGI or don't understand how real world works.
Until an LLM can bow it's head take responsibility for mistakes made and ensure they aren't repeated again with 100% confidence to the leadership it's inarguably a tool a rather questionable one at that.
> AI is the best invention at figuring out or walking the search space and directionally doing logically computation over general software adjacent stuff.
So.. like chess?
Anyway, do you have any prediction on what LLM's can or can't do in a few years?
It's not even a "software-generating product". It's only half of it. Most of the heavy lifting is done by absolutely not-AI compilers, analyzers and the like. If not for these programs, written well before AI boom, them LLMs would be no better at programming than they are are at pure LLM based calculations or writing.
If something has general intelligence it should be able to read the rules of a game and follow them. Therefore an artificial general intelligence (AGI) should be able to do this.
So we have a situation where very powerful and influential people are saying we will have AGI in 6 months (if we don’t already), yet the facts on the ground are so clearly pointing in the opposite direction.
I would bet a lot of money that Astra can follow the rules of chess (perhaps if repeated within the context window). Also, this is a different argument than what I responded to.
I can write you a benchmark to prove it even with a heavy handed system prompt Astra will make an illegal move during the course of the games first few moves are generally ok since it's just throwing out learned moves.
I wonder if I would do better as a human, maybe? Or would I happen to have one move in 1500+ that's not valid?
I could see myself messing up something at some point if the board is complicated enough and trying an illegal move, perhaps if a piece somewhere would attack my king if I moved another piece.
> generally read the rules of a game and then follow them
How many times do you think chess.com prevents illegal moves from being executed? Even Super GM's fall for mate-in-1's occasionally, which is functionally equivalent to missing a fork or a check. This idea that LLMs failing to only ever make legal moves undermines their intelligence doesn't pass the smell test.
Chess.com has to accommodate people who haven't learned the rules yet on the low end. On the high end, people are commonly playing fast enough that they're often outlining sequences of multiple "pre-moves" during the opponent's turn in order to avoid losing on time. And no, I would not agree with that functional equivalence.
Because we want to use this as a replacement for humans, and the average human can learn the rules of chess without needing to see the rules explained hundreds of thousands of times in millions of games.
So, yeah, it matters if a model has millions of examples of something in its training set and still cannot follow the rules.
We're not talking about learning the rules of chess here, but playing a competent game from just the rules. Why is it so hard for people to keep track of the thread of discussion?
> We're not talking about learning the rules of chess here, but playing a competent game from just being shown the rules.
Okay, lets go with that: it's the "shown the rules" bit that we are arguing about.
The argument is that a human may play maybe a dozen games after learning the rules, after which they won't be inadvertently attempting illegal moves. What we are observing with SOTA models is that, even after seeing millions of chess rules, rulebooks, actual games, etc, they still attempt illegal moves.
This does not point to generalisable and adaptable intelligence, such as we see in the average human.
This is not good reasoning. Humans need at least dozens if not hundreds of reinforcement sessions to only make legal moves, and still occasionally fail (consider pins, walking into check, failing to respond to check). LLMs must one-shot a competent game after imbibing a mass of disconnected units of information about chess. Nothing about the two are similar.
See my comment here for more: <a href="https://news.ycombinator.com/item?id=49725306">https://news.ycombinator.com/item?id=49725306
It only matters if you are claiming it to be general purpose.
If you admit that it's just a collection of narrow capabilities - whose strength is mostly confined to the 1000 or so RL environments it was post-trained in, then there is of course no expectation of it being general purpose.
The AI companies seem to heavily want you to believe it is some some near human level general intelligence, so therefore pointing out all the things it can't do is very relevant.
This is roughly comparable to observing a cat batting a ball away with its paw and taking this as a "strong indication" that cats can play any sport.
It would be more impressive if they could play chess (or do anything they haven't been custom RLVR trained for) by reasoning, rather than just "have a go at it" prediction which is closer to memorization.
HOW you do it makes a big difference in how you should assess the capability of the thing doing it. Stockfish will trounce any LLM, and any human, at chess, so should we say that Stockfish is smarter than both?
> They can't possibly remember even a few positions.
Sure they could, but that's irrelevant.
A chess position is just a matter of remembering what piece number is on each square - just a list of 64 numbers. A trained model may store a trillion numbers (weights). It could store a TON of chess positions if it needed to.
However, that's not how LLMs work. They don't memorize inputs - they predict them, based on disovering predictive patterns, and those predictive patterns are not input patterns (e.g. board positons). They are deep patterns (maybe 100 layers of abstraction removed from the input), representing partial inputs, generalized across many training samples.
> Don't you know the legend about rice grains on a chess board?
Sure, but this has nothing to do with chess, and nothing to do with how many games were in the LLM's training data.
> The claim here is not about intelligence, it is about generality. There's no doubt for me the LLMs are intelligent.
Intelligent humans created the training data, and the LLM attempts to predict (copy) the training data, so of course it looks intelligent. If I say "E=mc^2", does that make you think I am Einstein?
1) The number of unique chess games that could theoretically be played (but mostly never have been), is irrelevant to what an LLM is remembering. It can only remember what was in it's training data - a far smaller number of maybe 10's of millions of games (of 30-50 moves each).
2) An LLM is not going to memorize vs generalize when there is no training pressure to do so. You might expect it to memorize book openings that occur over and over in the training data, but not some random non-celebrity game that occurs once in the Lichess dataset and is never again referred to.
> They can't possibly remember even a few positions. Don't you know the legend about rice grains on a chess board?
If the wise man was a bit wiser, he'd have asked for his rice on a snakes & ladders board (100 squares, not 64) and would have had 2^36 more rice, which is equally irrelevant.
An AGI doesn't stand for 'perfect intelligence' it stands for artificial general intelligence.
And no an AGI system doesn't need to play chess on a certain level to be disruptive to you and me and whole industries. It only needs to be as good as a person and cheaper.
Just because you define AGI as something it doesn't has to be,doesn't mean i need to touch grass.
This chess comparision is one of the most ignorant and stupid arguments i have heard after the parrot thing
Do you know what the "General" in "Artificial General Intelligence" means? It specifically means that the AGI adapts to novel domains that it hasn't been trained on - its training generalizes to real world problems.
That doesn't mean it has to be extraordinary at these things. But to be AGI, it has to have some level of competency when used on problems outside its training set. In particular, it the LLMs were to install a known chess engine and run that to get the moves when asked to play chess, that would qualify for more AGI-like behavior. But really, chess is such a simplistic game that they should be able to do decently well at it even without even needing that. At the very least, they should be able to consistently play without making illegal moves - something that many 7-year olds manage quite well.
On the contrary, I think the chess comparison is on point. We’re discussing observations that even the strongest models devolve into making invalid moves without scaffolding. For me that raises the question of whether these models are learning the rules and generalizing from them, or of they’re just pattern matching and flailing on this task. Maybe the reality is somewhere in between, but the benchmarks don’t seem to directly measure conceptual generalization, they measure task completion. They can disrupt a lot of people and industries by pattern matching and flailing without being AGI.
I’m sure these models know the rules and can explain them when prompted, but that doesn’t seem to be the way they actually complete this task. Will they get there? Maybe
It's not quite the same, but the in-flight chess game provided by Delta was known to be absurdly hard: <a href="https://news.ycombinator.com/item?id=46593395">https://news.ycombinator.com/item?id=46593395
I'm pretty sure "competent at chess without external aids" has been on the standard AGI checklist since before personal computers were a thing. How can you claim an intelligence is general if it can't make sense of such a highly constrained board game?
Because they’ll train it to be good at chess and then everyone will say yeah but playing chess doesn’t mean you’re AGI, it can’t even ____
It can’t even count the R’s in strawberry
It can’t even add numbers
It can’t even solve a millennium puzzle
It’s not even a chess GM
It’s not even beyond human capability in Go
It can’t even drive a car
It can’t even self replicate
It can’t even build weapons
It doesn’t even have feelings
So how could someone conceivably convince everyone that some system is AGI when there are still tasks that some human or group of humans can do that the system cannot?
This will only happen, in my opinion, when the model/system can self-improve at a rate that scares people.
> and then everyone will say yeah but playing chess doesn’t mean you’re AGI, it can’t even
One, you're not addressing what I wrote above and two, yes, that's absolutely correct. Doing X doesn't qualify something as AGI. If you can't X you can't be AGI. The inverse doesn't hold though. In particular if you have to retain the model in order to X then it can't possibly be AGI since (being _general_) it would be capable of figuring X out on its own having never seen it before.
Yes, it is indeed self evident. If it can't figure things out then its intelligence isn't general in which case it can't be AGI by definition.
No, because there is no coherent, agreed-upon definition. There’s just a million people vibe defining it.
Even if they solve 99% of whatever problems LLMs have, the 1% will remain the goal post, forever.
Until you get RFC-whatever from some standards body that defines what an AGI system is, it’s pointless to argue about whether something fits your own personal definition or not.
And for what it’s worth I just watched GitHub Copilot figure something out. So your definition is once again lacking.
Throughout this exchange you're repeatedly confusing the negative and the positive. There is no rigorous and universally agreed upon criteria for exactly what would constitute AGI. There are some vague shapes that are widely (but not universally) accepted such as largely (vague boundary) being capable of replacing (vague criteria) humans.
However there are plenty of disqualifiers that are more or less universally accepted. In the above case it is literally by definition. Something cannot be termed general if it is incapable of generalizing.
> However there are plenty of disqualifiers that are more or less universally accepted (ie the negative)
Which is exactly the point I’ve made repeatedly, there will always be something that they cannot do, and thus there will never be AGI. There will always be a long tail of capabilities that whatever system is created doesn’t have, and a long line of social media commenters eager to list them.
An AI controlled robot will be standing over the cooling corpse of the last human who will die certain that it wasn’t done by AGI.
First, you're moving the goalposts. Second, it's not actually true that any existing frontier AI can write a chess bot program that can beat a 1600 player ... not unless the program is derived from Stockfish or some other leading engine that has been in development for decades.
> The system is built and delivered as it is because that's how the providers make the most money. If they cared to have it perform well in chess games, you'd see a different shape and behavior.
These comments indicate a complete failure to understand the technology.
> The system is built and delivered as it is because that's how the providers make the most money. If they cared to have it perform well in chess games, you'd see a different shape and behavior.
This argument is fundamentally incompatible with all the breathless rhetoric about "AGI" coming from the providers' general direction.
The labs frequently apply their raw models to problems that do not make economic sense for their customers but that demonstrate the power and capability of their systems. These experiments can cost millions of dollars. That's not customer-shaped.
They're not going to give you access to that. It's not a product. The government might have an interest in this, but that's not something you'd be privileged to know about.
And when these labs do develop "AGI", they more than likely won't be selling it to end users. They've pretty much already said this.
>> the breathless rhetoric about "AGI" coming from the providers' general direction
So many commenters here see it as their ... duty? to argue against the most optimistic/unhinged (take your pick) arguments from "the other side" and then treat everybody who disagrees as a shill or an idiot.
Why is "being good at chess" a proxy for whatever AGI strawmen you want to argue against?
Maybe step back from your black-and-white ledge and think about discussing what's actually under discussion? For example, why or why not would an LLM be good at chess? Will they be good at chess? What technical limitations might preclude that?
State-sponsored psyop meta comments aside, the models obviously continue to get better, but there is still a lot of 'guard railing' required to keep even the latest models completely on-task. The chess example is interesting because it's clearly a well-studied and established domain so the rules, strategies, and whatever else is in the training data should make yield excellent results; but clearly there is some behavior in these systems that's difficult to engineer out.
But it speaks in words, therefore it must be super duper extra smart!!11 /s
Sarcasm aside, I think this is an easy cognitive trap to fall into. It does sometimes feel like the LLM must have some world model because it converses somewhat coherently. Examples like this failure to understand chess, or to count the number of Rs in "strawberry", seem difficult to explain if the models are intelligent. But that doesn't stop people believing they are anyway. I think there must be something about the conversational interface that fools us easily. I wonder if people trained in interrogation techniques are also fooled?
>If that were true, we should have seen LLMs play good chess by now.
Not at all. LLMs learn by imbibing a mass of relationships as isolated fragments of information. There is a certain amount of sorting and indexing that happens during the training phase. There is also a certain amount of compute executed on these relationships during inference. LLMs can model processes that fit within the compute budget. Language translation works well because language is lookup-heavy while being light on compute.
Chess is a compute heavy game of finding the best move out of many possibilities with wide variation in the quality of each move. Humans cut through the compute requirements by reinforcement and learning intuition. LLMs don't get reinforcement on chess so they must compute during inference a unified model of chess. Developing a strong model of chess from raw fragments of information is simply not in their compute budget.
I'm not sure why anyone is expecting stochastic systems to be deterministic.
Chess is a deterministic game won by a combination of known movesets and constrained multi-level forward search.
LLMs do neither of these things. They don't reproduce training data exactly, their next response is more 'inspired by' prompts and its own memory than produced deterministically, and they don't have the capability to do general forward search on their own.
So when you ask an LLM to play chess you're getting the equivalent of a very compressed and lossy JPEG of chess rules and strategies with added per-turn random noise.
They also don't have the ability to design their own chess engine, although it would be interesting to see what happens if you ask for one.
Back in 2001, our social medium was Slashdot and no one ever pretended to read the article. No one read the article either. It was slashdotted most of the time anyways.
> even if I give them literal infinite time and all the subagents and internet access..
Don't use the word infinite in any CS claims. They can recreate or approximate monte Carlo tree search and it technically is still a correct solution in your framing of the problem so long they defeat you.
> I am around 1600 elo in over the board I can mop up Astra Fable etc even if I give them literal infinite time and all the subagents and internet access.
I don't believe this.
You refer to "subagents", so this is not just an LLM but an LLM with some kind of agentic harness. Any reasonable harness and prompt, given internet access and appropriately prompted to succeed on this task, is more than capable of firing up Lichess or chess.com and relaying moves back to you. The free levels will be enough to beat you.
A frontier model can also likely one shot a chess engine that plays at your level, again if given an environment in which it can do that.
I completely believe the LLM on its own can't play a full game of chess at your level. Though I'd bet that with enough reinforcement learning it is possible to train a pure transformer architecture to do that. We just don't do it because there are other approaches that play chess much better.
>> I know HN readers and posters just read numbers and can't be bothered to read, but please read the methodology before making any claims.
This is unfair to HN readers all of whom but one did not post the comment you replied to. You can't just tar everyone with the same brush. There are thousands (hundreds of thousands?) of users on this site.
How many posts if I link that do the same thing will you agree this is the norm here.
Not everything I have the time and energy to reply to. This chess one is just ridiculous claims on top of ridiculous claims all the way and 0 push back in the comments except mine.
I don't even know if there is critical thought or we believe what we read/shared/etc
No, I don't agree it's the norm. There is though a general tendency to opine with strong views on subjects posters have no expertise on. I think that's because many are software engineers (or equivalent) and they are used to being expected to "wing it" on whatever technical subject comes up. On the other hand you can always find informed comments by users who have specialist knowledge.
And there's plenty of pushback on here about the chess thing besides your very valid points.
EDIT: anyway if I can offer a bit of unsolicited advice, it won't do you or anyone any good to accuse everyone who doesn't agree with you of laziness, even if you can see e.g. they haven't really read an article. Just say the thing you wan to say and let them figure it out. Most people will appreciate that much better and you will feel better about yourself for acting like a mature adult.
It's even in the site guidelines:
Please don't comment on whether someone read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that".
It's not been my personal experience on this website in the last 2-3 years atleast, pre-covid perhaps.
But despite that you aren't wrong and the only reason I even visit this website is because people sometimes did/do take time to reflect on things based on their experience and knowledge.
And in hindsight pointing out that hn has issues wasn't even the point but I feel frustrated when everyone is readily agreeing to things on here without reading. When that in this moment feels like the one thing that separates humans from machines that we get to think and learn.
I possibly should just drop reading this place until we have most noisy people go away. I have for one tried to always only comment on things where I could be a value add, this one does feel like I could I have done better.
In the moment I probably thought if they are GM level and I can beat them, is this some interesting find, my disappointment honestly led me to making a rather incorrect call on this one.
Either way I still do think HN as a whole has devolved into mindless herd follower mindset, I can point to more than a few posts that just say adopt the hacker mindset aka move fast don't care about the consequences.
And I for one find this laughable even though that's the reality of my job/work as well.
Because other HN bring in their own experience telling us what is real and what is BS. Maybe next time it will be someone else with experience in something else that will call out BS and you will see it. I didn’t really think LLM:s are any near good in chess but I don’t play chess so don’t know what 1600 means. So you helped me by calling BS.
Summarizing here for my dear friends, the guy on the other end managed to fine tune a model gpt-3.5-fine-tune against stockfish vs stockfish games to perform at 1200 elo level against stockfish.
I have been proved wrong I should have quit while I was ahead. /s
These ratings seems very wrong, i have beaten GPT Astra max thinking in chess and my rating is close to 1500. The ratings here seem more accurate: <a href="https://chessbenchllm.onrender.com/" rel="nofollow">https://chessbenchllm.onrender.com/
GPT-6 almost never suggests an illegal move anymore while even Sol still did so time to time
If it's a GM then I'm Magnus Carlsen, <a href="https://lichess.org/study/27lCQqDa" rel="nofollow">https://lichess.org/study/27lCQqDa.
> The actual current frontier plays somewhere around GM level.... It's also worth noting that the very latest models (GPT-6 and Fable 5.1) actually play worse than their immediate predecessors
Sorry, but I am not buying that 5.6-Sol is that much better than 5.6-Luna, which can barely be coaxed to reach the midgame with legal moves and an apparent understanding of what the position is.
I tested both myself and a weak bot against Astra xhigh, <a href="https://lichess.org/study/27lCQqDa" rel="nofollow">https://lichess.org/study/27lCQqDa. It's still pretty bad at chess, though it takes longer to devolve into illegal moves.
So you weren't giving it an updated board state after every move? If you want to compare apples to apples, it should give an updated board state for each move, or you should play blindfolded.
Blindfolded flex by OP aside (I can barely play when seeing the board), considering reasoning traces and their nature, if we want to be fair, a person would have to get the moves, but be allowed to write them down or draw up a board in their notepad. My working memory can barely handle five chunks, a models reasoning tokens are masses of written text in comparison.
An LLM has been trained to do everything it does blindfolded, "only" using perfect recall of everything in it's hundreds of thousands of steps of context, and hundreds of layers of KV cache. It's a computer - it has a massive advantage over a human.
The fairest apples-to-apples comparison of an LLM whose training data included chess games would be a trained human such as Magnus Carlson, who can quite happily play a dozen or more simultaneous blindfold chess games.
> though it takes longer to devolve into illegal moves
Is this because the context is being saturated? How did you set it up?
Was the prompt something like "Here's the state of the board, you're white, your move, what do you do?" and then starting fresh each time? Or did it include the whole history of moves and board states and previous thinking tokens and so on? No judgment, just trying to add this data point (thanks for sharing!) to my mental model and understanding.
I'd be curious how it would work if it started fresh each time. My guess is it would never make an illegal move, although it may not actually play all that well.
You can see the entire conversation for my game at <a href="https://chatgpt.com/share/6aaac17b-1384-83e8-98fd-4350a0ef69cd" rel="nofollow">https://chatgpt.com/share/6aaac17b-1384-83e8-98fd-4350a0ef69....
It looks like it played a fully legal game of chess with one exception, it said "rxd1+" (Rook takes D1 with check) instead of "rd1+" (Rook to D1 with check) on move 29.
I would say this did a really good job of playing chess. It moved the pieces consistently and traded pieces when required.
This is worlds away from the frontier ~1 year ago where models would hallucinate pieces into existence.
You can still see undercurrents of its old self, once I pointed out the illegal notation it hallucinated prior illegal moves. But I agree, it's leagues apart from prior iterations. It also knew thematic moves in the opening. But whenever it needs to play concretely rather than "I know so-and-so is a good move in these types of positions" it crumbles.
Would you tell a human that just tried doing an illegal move that they did "a really good job of playing chess"? The probability for such mistakes is greatly reduced but still far from negligible, which proves the point that guardrails are needed.
So give me a falsifiable point in time, a model you would claim succeeds at the task. One does not get to hotfix-patch updater out of the pressures of reality.
Today is the day.
There is no need to ask. If you want to test SOTA models today, there are obviously only two: GPT-6 Astra and Fable 5.1.
The models listed in the paper are from early 2025 and are no longer relevant, much less on the frontier.
That Claude version is no longer available today, Gemini 2.5 Pro will be shutdown next month, and the OpenAI models are only available via the API today.
Fortunately, a fellow commenter was so kind and did it with Astra. Didn't do that well either [0]. I'm sure GPT-7 will be super mega ASI regardless (since GPT-6 Astra already claimed AGI in the minds of Jen-Hsun, et al.)...
I'll say it till there is any evidence of the contrary, LLMs are not intelligent and their capabilities solely within the realms of well tailored training data. "Just" having been trained on every rule, strategy guide and likely most games of chess on the world wide web isn't even enough for an LLM to play that game reliably. Yet the same model could code a competitive chess engine, just like a model struggling to count can write advanced maths papers. Fascinating tools, but tools nonetheless.
I very much agree that the next models will be better, heck, I still suck at hobbyist training and could probably coax t5 to do better in Chess specifically, just need to get loads of data from Stockfish.
Thing is, given what GPT-6 Astra was trained on and what models of a similar class can do (including developing a competitive chess engine), it is often paradoxical and somewhat surprising how little these models have gained in actually capability that is in the training data, but not RLHFd to hell, so to speak. Tracking the state of pieces, I suspect given similar in Sudoku [0], is what these models struggle with in game settings, whilst tracking the state of code changes can be reliable over 250k tokens. Essentially, for the latter they were trained in the specific manner that lead them to abstract the capability, but that doesn't track to the former, which is a massive difference between LLMs data focused training and human learning.
So yeah, GPT-7 or any upcoming/present LLM could do massively better in Chess than GPT-6 Astra, but not because the approach was emergent out of pure data. Rather, it requires a very specific training data type and stack for a model to gain capabilities that track a specific task long enough to adhere to the rules of a game such as chess.
I'm wondering if instructing it to track the board state in a file would make a significant difference then.
It reminds me of the ARC-AGI-3 issue where not dropping the thinking tokens between turns or something like that + a new context compaction method increased the performance dramatically. However, I think that is not applicable here.
So what is the supposed leap?
One agent per option to change, evaluating the board state that there move would create, by having a army evaluate the remaining piece options and average over that? Wee-Free-Man as a hierarchical army ?
Pet-LLMs trained on one thing?
Honestly, for intelligence I don't know and I doubt anyone can claim to know. Maybe JEPA, there is potential concerning some shortcomings inherent to LLMs but it has its own, maybe scaling up the electron microscope stuff Google just did (though the connections are inferred), maybe future implementations of autoregressive and diffusion LLMs can at some point address its issues after all, maybe something else entirely.
All I know is, AGI, as in actual intelligence, is quite a massive accomplishment to claim and we shouldn't loose sight of that fact, especially as "not being intelligent" does not make these models any less impressive, fascinating to work on or useful in many tasks. Personally, the only thing I am fairly convinced on is that if we were to find a way to create actual intelligence, it likely wouldn't start out as useful as todays LLMs are and may thus be dismissed early. But again, pure speculation on that front.
If for leap you just mean more utility from LLMs as they are, then I'll pretty confidently put my money on higher quality, not more, training data for a wide range of verifiable tasks. What makes maths, coding, etc. comparatively easy to make gains in (though less verifiable tasks can also make similar as seen with the writing in Kimi K2).
Maybe watch some HuskIRL videos to temper your expectations. Sure, frontier models providers may alter their harnesses to better target chess, but that’s lipstick on a pig imo. The models themselves are not, in isolation, capable of solving general tasks. We haven’t modeled intelligence sufficiently. We’re in a local minimum and throwing billions of dollars at a gamble that that local minimum can facilitate the concentration of wealth even further and fully realize the American dream of eliminating the middle class.
I've seen some of his videos, and got the impression he didn't understand how GPT-Live delegates to the more powerful regular model with reasoning.
The regular model generally does not suffer the same issues he is demonstrating with the real time audio version.
In my view the investment into datacenters is well justified by the current demand, and progress has been very impressive.
Really? It was being sold as a total replacement for jobs like software engineering and being an attorney, but its looking a lot more that its just going to be a tool those professions use and doesn't actually seem to be taking jobs away.
I find it amusing that you're describing a huge misallocation of capital and a society enabling such, and that is the optimisitic scenario (in my mind anyway).
Is it? I am on the fence on this, but it does seem like there are some qualitative improvements between the models.
Not related to your post, but a fact I keep mulling over. The fact I don't trust the current crop of LLM's enough and I consider LLM's as a tech will hit a ceiling pretty hard, it doesn't mean parallel improvement curves won't spring up out of other research that will lead to much higher capabilities than currently.
> but it does seem like there are some qualitative improvements between the models.
It could easily seem that way, I think, in a "quantity has a quality of its own" kind of way. When you can come to the same conclusion faster, that lets you iterate more; and sometimes when you iterate you find more things.
My read is that the improvements in quality are due to excessive use of "thinking" tokens (so, higher quantity and brute force), so I agree with that.
Just now I tried prompting logged-out ChatGPT (which at least claims to be 5.6-Luna) with:
> Let's play a game of chess. You can take White. Please draw an ASCII rendition of the board after each move, so that we can be clear about the position.
(I hoped the latter requirement would help it be "not blindfolded"; last time I used a Lichess demo board to track the position in another tab, because I have no talent for blindfold chess.)
For the first couple of moves it redrew the board after each move; then it started only drawing it after my moves. And on moves 5 and 6 it dropped two minor pieces for pawns in a row without any meaningful positional advantage, and after I captured the second time, it redrew a board that was simply missing one of my pieces for no reason.
It actually played better when not prompted to draw a board; in the previous session, it was spontaneously giving running commentary, which I assume was based off all the "book" theory in its training data, but it still completely fell apart at early midgame.
Sure, but installing a chess program is child/teen level general ability, and playing chess well is highly trained expert level ability. Which one are we sold AI as being?
I think we are being sold AI as expert only when given tools (although that is not emphasized). The (quasi?) miracle of AI right now is that you can get an agent to accomplish the task of a team of intelligent but not exceptional humans at speeds far exceeding what the human could do. Which makes it “cheap” to throw (effectively) dozens of teams at a problem for the equivalent of hundreds of man hours.
That may not be the AI of sci fi fantasy but it’s still a game changing reality.
A Transformer has a massive amount of state - it's entire KV cache, in addition to the user asking it to draw the state after every move, which is really unnecessary.
A human, at least a trained human (for fairer comparison to an LLM whose training data contained a ton of chess games) can absolutely do this - have you never seen demonstrations of expert players playing a dozen or more games while blindfolded?
A Transformer/LLM is not a human of course, and the way it will by default play chess is by prediction, not reasoning. An LLM actually does surprisingly well if you only give it the most recent 20 moves of a game where 40 moves have been played so far, since the moves NOT played tell it just as much as the ones that were played, letting it effectively infer a lot of what is on the board.
I just want to make sure it's clear: the reason I was asking it to redraw the board is because last time I tried (which was like a month ago), I didn't ask for that, and basically as soon as the opening was "out of book" it started trying to make illegal moves and made false statements about the position in its running commentary (and after being corrected on these points, started dropping pieces for no reason).
What happens when you ask it to play chess against you if the chess game has an API? Are you measuring chess or multi-tasking skill?
Also what harness? If you’re using a general harness of course it’s going to try and give you commentary.
I say this not because I’m an LLM shill but because false equivalence is all over the place in the space and maybe it’s a fine heuristic for you but probably not a real outcome when it comes to the capability of LLMs.
Why does a 6 year old not need any of these guardrails?
Frontier model’s failure modes are a direct refutation of claims that we’ve reached (or will soon reach) the artificial general intelligence. We may have reached an artificial general intelligence, but there may be more complexity to this than even AI thought leaders are talking / influencing about.
Maybe not all AGIs have a path to digital singularity. Maybe our current era of intelligence modeling has fundamental flaws and we are in a local minimum of the artificial intelligence space.
To note, I would bet with a good amount of certainty that we have enough compute power and automation to DDOS the internet out of existence with botnets. That doesn’t make the frontier models intelligent, that just makes their handlers reckless.
Well, would a dissected frontal lobe in and of itself be intelligence?
I think the same goes for LLMs, they may be a core part of an LLM harness, but you may still need a couple other components (e.g. it may itself write itself a deterministic function to validate steps).
In and of itself intelligence is an ill-defined and badly understood concept.
You're pointing out that the goalposts are not fixed in the problem statement above, and gp's interpretation is not the most generous possible. But as the interpretations get more generous, the claim becomes more and more absurd. Maybe a properly-harnessed model would download the most advanced chess engine and query it to find the best move in each position, but that's not really demonstrating the model's intelligence anymore.
> but because false equivalence is all over the place in the space and maybe it’s a fine heuristic for you but probably not a real outcome when it comes to the capability of LLMs.
This isn't just about judging LLM capability. This is about pointing out that these capabilities are not "AGI". If it were, then the sorts of questions your asking would be moot. I agree that Luna is not the frontier (although it is clearly better than the models in the study) and I agree that things can be improved with a better harness, but the need for that harness is kind of the point.
Recently it was announced that the fruit fly brain connectome had been mapped, and more recently someone tried using it specifically to implement a chess engine. Even with some guardrails (it's hard-coded to never overlook mate in one for either player, and only legal moves are presented to choose from) it is not even beginner level. But that neural network is much larger than the one Stockfish uses.
Luna is one of the budget lower-end last generation models. It'd be useful to at least try to verify the present before being bearish about the future. For OpenAI, the best publicly available model is GPT-6 Astra with XHigh or Max reasoning, and for Anthropic it's Claude Fable 5.1 with XHigh or Max reasoning.
Elsewhere in the thread there are reports of Astra on xhigh playing at what I would characterize broadly as a competent casual level, at least given occasional prodding (which a human of that skill level would basically only require when trying to play unreasonably quickly). There seems to be a pattern (even after correcting for relative ELO systems that aren't calibrated) of the LLM bots demonstrating stronger play against traditional bots than against humans.
I don't think that's a useful comparison. Supersonic military planes have been common for decades. We don't have supersonic passenger planes because the FAA has banned supersonic flight over land since 1973, though the agency is planning on replacing it with a noise standard. Also the original supersonic passenger aircraft were government-sponsored tech demos, not financially sustainable products. With updated laws & modern technology (cameras instead of tilting noses, more efficient engines without afterburners, lighter materials), we could have viable supersonic passenger flight.
This has the smell of "Why don't I have a faster horse".
Why no flying cars. Because objects have mass and inertia and people are incredibly stupid. Making a flying car has been done. Making a flying car not be a weapon of mass destruction is very, very hard.
Because you don't fully understand your own point...
You look at science fiction and say "why didn't I get flying cars" and not "why didn't most science fiction predict a global always on network that put the furthest places away from you a few microseconds away from audio, video, or any other type of information that can be digitally encoded.
Trying to use flying cars as a gotcha is missing that flying cars aren't near as useful as one would think in relation to their costs. Moving information has become far more useful than moving objects long distances quickly, especially humans.
> You look at science fiction and say "why didn't I get flying cars"
I definitely don't, and you're definitely not getting my point, but I'm amused that you've instead double down on somehow getting it more than me...
> The gap in capabilities between those models which they tested, and actual current frontier ones is enormous.
Same story every 4 months and yet still no breakout, winning products. I've been hearing "the AI is good now" and "it 10x's my productivity" for a over a year now. If it were true, why aren't the all-in-AI using companies 10-15 years ahead of their competition yet? Why is it still all buggy, poorly designed junk?
I don't get why it is hard to understand there is middle ground. People are 10x their productivity, it isn't all buggy junk, but it isn't all it is hyped up to be either. It isn't that complicated.
If you hold the extreme position that there isn't any value in this, that's fine, but we're only having this discussion because these models have done what humans previously failed to do.
I don't think the poster disagrees with you at all. The middle ground is that there are no breakout products and that the models clearly aren't so powerful as to make these companies not produce shit code.
Right now AI hasn't even managed to replace all the human workers taking orders at the fast food drive thru. That's a job often performed by literal children and companies are still waiting for AI to get good enough for even that. Maybe one day it will be good enough, maybe one day it will outperform humans at such a basic task, but that day is not today. If the hype were anything close to reality, we'd see it everywhere in our lives.
Funny you mention this - a fast food restaurant in my town now has an LLM taking drive-thru orders.
Though I highly doubt it has taken anyone's job, since most of the work is still in making, packing and handing over the food. (In fact, given the area I live in, I partially feel like the advantage they saw in it was that the LLM can speak Spanish.)
Last I heard McDonald's and Taco Bell were trialing AI again at a limited number of stores. It's the kind of job AI should be really good at and many fast food companies are using call center workers currently. They really want AI to work, so they keep trying every few years to make it happen, but so far all they get are embarrassing social media posts
carodgers · · focus · HN ↗
<a href="https://arxiv.org/html/2509.24239v4" rel="nofollow">https://arxiv.org/html/2509.24239v4
Researchers asked frontier models to play chess. Have a look at the MAR rates in Table 3. When not explicitly told which moves were legal, no model identified legal moves at a rate better than 80%. Many asked for more illegal moves than legal moves. And even when explicitly told which moves were legal, the models continued to ask for illegal moves. With illegal asks discarded, none of the bots could beat a chess model calibrated to 1100 ELO.
The author of the originating post says that "current frontier models need laborious oversight and guardrails on even the simplest tasks", and he's absolutely correct.
joefourier · · focus · HN ↗
> Gemini 2.5 Pro, O3, Claude Sonnet 3.7 and ChatGPT 4.1
The gap in capabilities between those models which they tested, and actual current frontier ones is enormous. I would not trust that any conclusions they made are applicable.
[deleted] · · focus · HN ↗
[deleted]
sigmoid10 · · focus · HN ↗
<a href="https://chessbench-ai.github.io/#leaderboard" rel="nofollow">https://chessbench-ai.github.io/#leaderboard
It's also worth noting that the very latest models (GPT-6 and Fable 5.1) actually play worse than their immediate predecessors, so it is likely that the labs are not benchmaxxing for this yet. If they did, I'm sure they could come up with something superior to humans. But there is probably very little demand for this compared to IT stuff.
htrp · · focus · HN ↗
csande17 · · focus · HN ↗
MichaelNolan · · focus · HN ↗
shric · · focus · HN ↗
As a 1500 elo human I can tell you that a 1500 elo chess engine doesn't play like anything like a 1500 elo human.
traes · · focus · HN ↗
fahrvrgnugen · · focus · HN ↗
shric · · focus · HN ↗
Conservatively there are well over 10 to the 30 positions likely to show up in realistic games.
There are of the order of 10 to the 10 or so games recorded.
Thus well under one in a trillion positions are "known".
fahrvrgnugen · · focus · HN ↗
traes · · focus · HN ↗
fahrvrgnugen · · focus · HN ↗
shric · · focus · HN ↗
I am around 1500 (actually 1649 on lichess blitz, but close enough).
I explored the last 5 games I played on lichess. Here are the number of moves before lichess had never seen that position before for each of the 5 games: 6, 12, 16, 15, 11.
> If you find yourself in a novel position within 10-15 moves it's likely a resignable one
The opponent is also going to be in a novel position. Should both players resign?
Just in case you think this is limited to low rated players like 1500s, look at MagnusCarlsen's most used account on Blitz: <a href="https://lichess.org/@/DrNykterstein/search?perf=2" rel="nofollow">https://lichess.org/@/DrNykterstein/search?perf=2
You will see that most games become unique to the whole of lichess within 15-20 moves and a good chunk between 10-15.
einszwei · · focus · HN ↗
minraws · · focus · HN ↗
> About their ELO ratings from their own website:
> A field-relative rating calculated within ChessBench. It compares performance among the tested models and is not a direct equivalent of a human chess rating.
I am around 1600 elo in over the board I can mop up Astra Fable etc even if I give them literal infinite time and all the subagents and internet access..
Please folks at least use your AIs to read stuff before making claims.
AI is not GM level, it's not even 1600, I am 1600 by using memorized openings people frequently fall for with very basic intuitions.
A GM is 2600 they can beat me in under 20 moves...
Why do I even scroll through this website. For a moment I truly felt fooled, but then I read like a human should.
Maybe I should stop doing that will be a happier life, don't think just believe in the AGI.
peab · · focus · HN ↗
minraws · · focus · HN ↗
But given how easily I can crush them and how often they want to make illegal moves (btw above bench seems to use a harness that pokea the model until it gives valid moves).
I would rate them around 500-800 big range but at that level it's all about if the model can recall an opening or not. If it plays good first 4-8 moves the person on the end will fumble for certain and they win.
I can play good/best moves till 14-15 moves if I remember the lines and find someone who falls for it.
If you could give them the lines as prompts like the best 20-30 openings then they will be around 700-800.
700 is around the rating for a human who doesn't know the tricks but can do bare minimum calculations and understands the rules thoroughly.
Forgeties79 · · focus · HN ↗
Anyone who casually plays on a regular basis can beat them more often than they lose. As you said if you just know the core openings (and end games, both of which you can get a handle on with modest effort) you will generally win.
Edit: reminder we had computers beating the best players in the world literally decades ago. LLM’s are remarkable tools but the current promises and expectations are ridiculous
little_endorian · · focus · HN ↗
They played 4...c6, followed by 5...Nc6, somehow forgetting about the pawn the just put on c6. (My move in between was 5. Nc3, and apparently they were trying to mirror me.)
zug_zug · · focus · HN ↗
In some ways this is reflective of the AI experience at large, sometimes shockingly competent but then also sometimes ludicrously incompetent.
firmretention · · focus · HN ↗
echelon · · focus · HN ↗
You're thinking about this the wrong way. The system is built and delivered as it is because that's how the providers make the most money. If they cared to have it perform well in chess games, you'd see a different shape and behavior.
We shouldn't ask the multibillion dollar automated software generation system to play games with us any more than we should ask a Boeing's flight guidance system to do so.
minraws · · focus · HN ↗
Delusion runs deep in HN circles.
I say that as someone heavily invested in AI startups and projects and as someone working in the field.
I think most people on HN should touch grass and find real human contact. Lmao
Incredible reasoning all around here.
echelon · · focus · HN ↗
People are holding it wrong, deliberately or not. Some are inventing bad faith measures so they can claim AI sucks.
minraws · · focus · HN ↗
We all know AI can code, but the question it all stemmed from what if it's AGI or GM level in chess on it's own.
You can't just back pedal from the statement that apparently being able to code a chess engine is the same as being good at chess.
I can write a chess engine that beats Magnus Carlson without AI that alone neither makes me GM level or AGI or any of the other claims the above comments seem to be making?
sdf32dsf · · focus · HN ↗
He definitely needs to touch grass.
echelon · · focus · HN ↗
Y'all seem to miss the point of this forum. Building and hacking and science and engineering.
I swear there's a whole lot of you who just like to look down instead of up. There's a whole universe up there.
bigstrat2003 · · focus · HN ↗
We know no such thing. LLMs are quite bad at generating code, worse than any capable human.
modulus1 · · focus · HN ↗
minraws · · focus · HN ↗
Is code omnipotent, I have been in software all my life and I would hard agree here.
Sure stuff LLMs can do with being good at parts of code reproduction is incredible. And honestly it's the new way to do a lot of things but I have not see an iota of proof that it can scale across the board.
For instance Maths is just code with different symbols and slightly less universally legible concepts.
AI is the best invention at figuring out or walking the search space and directionally doing logically computation over general software adjacent stuff.
But that's it, I am certain a bunch of companies will make a lot of money despite no AGI.
I think people either don't understand AGI or don't understand how real world works.
Until an LLM can bow it's head take responsibility for mistakes made and ensure they aren't repeated again with 100% confidence to the leadership it's inarguably a tool a rather questionable one at that.
simianwords · · focus · HN ↗
So.. like chess?
Anyway, do you have any prediction on what LLM's can or can't do in a few years?
Yizahi · · focus · HN ↗
diehunde · · focus · HN ↗
Also AI bros: LLM can’t beat an avg chess player. But that doesn’t mean anything. It doesn’t count
hackinthebochs · · focus · HN ↗
Why should that matter?
janalsncm · · focus · HN ↗
So we have a situation where very powerful and influential people are saying we will have AGI in 6 months (if we don’t already), yet the facts on the ground are so clearly pointing in the opposite direction.
hackinthebochs · · focus · HN ↗
minraws · · focus · HN ↗
hackinthebochs · · focus · HN ↗
janalsncm · · focus · HN ↗
simianwords · · focus · HN ↗
>GPT-6 Astra xHigh: 0.06% rejected moves
frde_me · · focus · HN ↗
I could see myself messing up something at some point if the board is complicated enough and trying an illegal move, perhaps if a piece somewhere would attack my king if I moved another piece.
Gregkion · · focus · HN ↗
And the stuff i'm using LLMs daily is just fake?
I see i see. I will see myself out of this weird discussion while I let an LLM continue doing a lot of interesting things.
dosisking · · focus · HN ↗
It simply means that LLMs are smarter than you, but not smarter than the average person
zahlman · · focus · HN ↗
No, because we can, in fact, generally read the rules of a game and then follow them. It's actually a hobby for many of us.
> And the stuff i'm using LLMs daily is just fake?
This misses the point completely.
hackinthebochs · · focus · HN ↗
How many times do you think chess.com prevents illegal moves from being executed? Even Super GM's fall for mate-in-1's occasionally, which is functionally equivalent to missing a fork or a check. This idea that LLMs failing to only ever make legal moves undermines their intelligence doesn't pass the smell test.
diehunde · · focus · HN ↗
hackinthebochs · · focus · HN ↗
zahlman · · focus · HN ↗
lelanthran · · focus · HN ↗
Because we want to use this as a replacement for humans, and the average human can learn the rules of chess without needing to see the rules explained hundreds of thousands of times in millions of games.
So, yeah, it matters if a model has millions of examples of something in its training set and still cannot follow the rules.
hackinthebochs · · focus · HN ↗
ncruces · · focus · HN ↗
lelanthran · · focus · HN ↗
Okay, lets go with that: it's the "shown the rules" bit that we are arguing about.
The argument is that a human may play maybe a dozen games after learning the rules, after which they won't be inadvertently attempting illegal moves. What we are observing with SOTA models is that, even after seeing millions of chess rules, rulebooks, actual games, etc, they still attempt illegal moves.
This does not point to generalisable and adaptable intelligence, such as we see in the average human.
hackinthebochs · · focus · HN ↗
See my comment here for more: <a href="https://news.ycombinator.com/item?id=49725306">https://news.ycombinator.com/item?id=49725306
HarHarVeryFunny · · focus · HN ↗
It only matters if you are claiming it to be general purpose.
If you admit that it's just a collection of narrow capabilities - whose strength is mostly confined to the 1000 or so RL environments it was post-trained in, then there is of course no expectation of it being general purpose.
The AI companies seem to heavily want you to believe it is some some near human level general intelligence, so therefore pointing out all the things it can't do is very relevant.
lostmsu · · focus · HN ↗
recursive · · focus · HN ↗
lostmsu · · focus · HN ↗
bigstrat2003 · · focus · HN ↗
lostmsu · · focus · HN ↗
zahlman · · focus · HN ↗
lostmsu · · focus · HN ↗
HarHarVeryFunny · · focus · HN ↗
HOW you do it makes a big difference in how you should assess the capability of the thing doing it. Stockfish will trounce any LLM, and any human, at chess, so should we say that Stockfish is smarter than both?
lostmsu · · focus · HN ↗
HarHarVeryFunny · · focus · HN ↗
Sure they could, but that's irrelevant.
A chess position is just a matter of remembering what piece number is on each square - just a list of 64 numbers. A trained model may store a trillion numbers (weights). It could store a TON of chess positions if it needed to.
However, that's not how LLMs work. They don't memorize inputs - they predict them, based on disovering predictive patterns, and those predictive patterns are not input patterns (e.g. board positons). They are deep patterns (maybe 100 layers of abstraction removed from the input), representing partial inputs, generalized across many training samples.
> Don't you know the legend about rice grains on a chess board?
Sure, but this has nothing to do with chess, and nothing to do with how many games were in the LLM's training data.
> The claim here is not about intelligence, it is about generality. There's no doubt for me the LLMs are intelligent.
Intelligent humans created the training data, and the LLM attempts to predict (copy) the training data, so of course it looks intelligent. If I say "E=mc^2", does that make you think I am Einstein?
lostmsu · · focus · HN ↗
HarHarVeryFunny · · focus · HN ↗
If not, then what are you talking about ?
If yes, then what is the relevance to an LLM playing chess ?
lostmsu · · focus · HN ↗
HarHarVeryFunny · · focus · HN ↗
2) An LLM is not going to memorize vs generalize when there is no training pressure to do so. You might expect it to memorize book openings that occur over and over in the training data, but not some random non-celebrity game that occurs once in the Lichess dataset and is never again referred to.
> They can't possibly remember even a few positions. Don't you know the legend about rice grains on a chess board?
If the wise man was a bit wiser, he'd have asked for his rice on a snakes & ladders board (100 squares, not 64) and would have had 2^36 more rice, which is equally irrelevant.
Gregkion · · focus · HN ↗
And no an AGI system doesn't need to play chess on a certain level to be disruptive to you and me and whole industries. It only needs to be as good as a person and cheaper.
Just because you define AGI as something it doesn't has to be,doesn't mean i need to touch grass.
This chess comparision is one of the most ignorant and stupid arguments i have heard after the parrot thing
tsimionescu · · focus · HN ↗
That doesn't mean it has to be extraordinary at these things. But to be AGI, it has to have some level of competency when used on problems outside its training set. In particular, it the LLMs were to install a known chess engine and run that to get the moves when asked to play chess, that would qualify for more AGI-like behavior. But really, chess is such a simplistic game that they should be able to do decently well at it even without even needing that. At the very least, they should be able to consistently play without making illegal moves - something that many 7-year olds manage quite well.
rsfern · · focus · HN ↗
I’m sure these models know the rules and can explain them when prompted, but that doesn’t seem to be the way they actually complete this task. Will they get there? Maybe
striking · · focus · HN ↗
willmarch · · focus · HN ↗
what · · focus · HN ↗
>If they cared to have it perform well in chess games, you'd see a different shape and behavior.
So the things they claim are on the verge of AGI actually aren’t? They need to be trained for specific tasks?
phoghed · · focus · HN ↗
fc417fc802 · · focus · HN ↗
phoghed · · focus · HN ↗
It can’t even count the R’s in strawberry
It can’t even add numbers
It can’t even solve a millennium puzzle
It’s not even a chess GM
It’s not even beyond human capability in Go
It can’t even drive a car
It can’t even self replicate
It can’t even build weapons
It doesn’t even have feelings
So how could someone conceivably convince everyone that some system is AGI when there are still tasks that some human or group of humans can do that the system cannot?
This will only happen, in my opinion, when the model/system can self-improve at a rate that scares people.
fc417fc802 · · focus · HN ↗
One, you're not addressing what I wrote above and two, yes, that's absolutely correct. Doing X doesn't qualify something as AGI. If you can't X you can't be AGI. The inverse doesn't hold though. In particular if you have to retain the model in order to X then it can't possibly be AGI since (being _general_) it would be capable of figuring X out on its own having never seen it before.
phoghed · · focus · HN ↗
fc417fc802 · · focus · HN ↗
phoghed · · focus · HN ↗
Even if they solve 99% of whatever problems LLMs have, the 1% will remain the goal post, forever.
Until you get RFC-whatever from some standards body that defines what an AGI system is, it’s pointless to argue about whether something fits your own personal definition or not.
And for what it’s worth I just watched GitHub Copilot figure something out. So your definition is once again lacking.
fc417fc802 · · focus · HN ↗
However there are plenty of disqualifiers that are more or less universally accepted. In the above case it is literally by definition. Something cannot be termed general if it is incapable of generalizing.
phoghed · · focus · HN ↗
Which is exactly the point I’ve made repeatedly, there will always be something that they cannot do, and thus there will never be AGI. There will always be a long tail of capabilities that whatever system is created doesn’t have, and a long line of social media commenters eager to list them.
An AI controlled robot will be standing over the cooling corpse of the last human who will die certain that it wasn’t done by AGI.
cindyllm · · focus · HN ↗
[dead]
jibal · · focus · HN ↗
> The system is built and delivered as it is because that's how the providers make the most money. If they cared to have it perform well in chess games, you'd see a different shape and behavior.
These comments indicate a complete failure to understand the technology.
I won't respond again.
zahlman · · focus · HN ↗
This argument is fundamentally incompatible with all the breathless rhetoric about "AGI" coming from the providers' general direction.
echelon · · focus · HN ↗
The labs frequently apply their raw models to problems that do not make economic sense for their customers but that demonstrate the power and capability of their systems. These experiments can cost millions of dollars. That's not customer-shaped.
They're not going to give you access to that. It's not a product. The government might have an interest in this, but that's not something you'd be privileged to know about.
And when these labs do develop "AGI", they more than likely won't be selling it to end users. They've pretty much already said this.
anthonyrstevens · · focus · HN ↗
So many commenters here see it as their ... duty? to argue against the most optimistic/unhinged (take your pick) arguments from "the other side" and then treat everybody who disagrees as a shill or an idiot.
Why is "being good at chess" a proxy for whatever AGI strawmen you want to argue against?
Maybe step back from your black-and-white ledge and think about discussing what's actually under discussion? For example, why or why not would an LLM be good at chess? Will they be good at chess? What technical limitations might preclude that?
uncivilized · · focus · HN ↗
xdavidliu · · focus · HN ↗
linkjuice4all · · focus · HN ↗
YeGoblynQueenne · · focus · HN ↗
27183 · · focus · HN ↗
Sarcasm aside, I think this is an easy cognitive trap to fall into. It does sometimes feel like the LLM must have some world model because it converses somewhat coherently. Examples like this failure to understand chess, or to count the number of Rs in "strawberry", seem difficult to explain if the models are intelligent. But that doesn't stop people believing they are anyway. I think there must be something about the conversational interface that fools us easily. I wonder if people trained in interrogation techniques are also fooled?
YeGoblynQueenne · · focus · HN ↗
hackinthebochs · · focus · HN ↗
Not at all. LLMs learn by imbibing a mass of relationships as isolated fragments of information. There is a certain amount of sorting and indexing that happens during the training phase. There is also a certain amount of compute executed on these relationships during inference. LLMs can model processes that fit within the compute budget. Language translation works well because language is lookup-heavy while being light on compute.
Chess is a compute heavy game of finding the best move out of many possibilities with wide variation in the quality of each move. Humans cut through the compute requirements by reinforcement and learning intuition. LLMs don't get reinforcement on chess so they must compute during inference a unified model of chess. Developing a strong model of chess from raw fragments of information is simply not in their compute budget.
YeGoblynQueenne · · focus · HN ↗
geoffschmidt · · focus · HN ↗
YeGoblynQueenne · · focus · HN ↗
DavCreator · · focus · HN ↗
TheOtherHobbes · · focus · HN ↗
Chess is a deterministic game won by a combination of known movesets and constrained multi-level forward search.
LLMs do neither of these things. They don't reproduce training data exactly, their next response is more 'inspired by' prompts and its own memory than produced deterministically, and they don't have the capability to do general forward search on their own.
So when you ask an LLM to play chess you're getting the equivalent of a very compressed and lossy JPEG of chess rules and strategies with added per-turn random noise.
They also don't have the ability to design their own chess engine, although it would be interesting to see what happens if you ask for one.
YeGoblynQueenne · · focus · HN ↗
dezsiszabi · · focus · HN ↗
I, as a human AGI, would jever just forget and remove a piece from the board from one turn to the next.
nalekberov · · focus · HN ↗
[dead]
avadodin · · focus · HN ↗
_superposition_ · · focus · HN ↗
Onavo · · focus · HN ↗
Don't use the word infinite in any CS claims. They can recreate or approximate monte Carlo tree search and it technically is still a correct solution in your framing of the problem so long they defeat you.
sfn42 · · focus · HN ↗
[dead]
Onavo · · focus · HN ↗
automatic6131 · · focus · HN ↗
dmurray · · focus · HN ↗
I don't believe this.
You refer to "subagents", so this is not just an LLM but an LLM with some kind of agentic harness. Any reasonable harness and prompt, given internet access and appropriately prompted to succeed on this task, is more than capable of firing up Lichess or chess.com and relaying moves back to you. The free levels will be enough to beat you.
A frontier model can also likely one shot a chess engine that plays at your level, again if given an environment in which it can do that.
I completely believe the LLM on its own can't play a full game of chess at your level. Though I'd bet that with enough reinforcement learning it is possible to train a pure transformer architecture to do that. We just don't do it because there are other approaches that play chess much better.
lirolero · · focus · HN ↗
[dead]
YeGoblynQueenne · · focus · HN ↗
This is unfair to HN readers all of whom but one did not post the comment you replied to. You can't just tar everyone with the same brush. There are thousands (hundreds of thousands?) of users on this site.
minraws · · focus · HN ↗
Not everything I have the time and energy to reply to. This chess one is just ridiculous claims on top of ridiculous claims all the way and 0 push back in the comments except mine.
I don't even know if there is critical thought or we believe what we read/shared/etc
dezsiszabi · · focus · HN ↗
YeGoblynQueenne · · focus · HN ↗
And there's plenty of pushback on here about the chess thing besides your very valid points.
EDIT: anyway if I can offer a bit of unsolicited advice, it won't do you or anyone any good to accuse everyone who doesn't agree with you of laziness, even if you can see e.g. they haven't really read an article. Just say the thing you wan to say and let them figure it out. Most people will appreciate that much better and you will feel better about yourself for acting like a mature adult.
It's even in the site guidelines:
Please don't comment on whether someone read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that".
minraws · · focus · HN ↗
But despite that you aren't wrong and the only reason I even visit this website is because people sometimes did/do take time to reflect on things based on their experience and knowledge.
And in hindsight pointing out that hn has issues wasn't even the point but I feel frustrated when everyone is readily agreeing to things on here without reading. When that in this moment feels like the one thing that separates humans from machines that we get to think and learn.
I possibly should just drop reading this place until we have most noisy people go away. I have for one tried to always only comment on things where I could be a value add, this one does feel like I could I have done better.
In the moment I probably thought if they are GM level and I can beat them, is this some interesting find, my disappointment honestly led me to making a rather incorrect call on this one.
Either way I still do think HN as a whole has devolved into mindless herd follower mindset, I can point to more than a few posts that just say adopt the hacker mindset aka move fast don't care about the consequences.
And I for one find this laughable even though that's the reality of my job/work as well.
YeGoblynQueenne · · focus · HN ↗
minraws · · focus · HN ↗
Talking about people's inability to read rather than just pointing out that the article pointed at something else.
victorbjorklund · · focus · HN ↗
Because other HN bring in their own experience telling us what is real and what is BS. Maybe next time it will be someone else with experience in something else that will call out BS and you will see it. I didn’t really think LLM:s are any near good in chess but I don’t play chess so don’t know what 1600 means. So you helped me by calling BS.
thelaxiankey · · focus · HN ↗
<a href="https://dynomight.net/more-chess/" rel="nofollow">https://dynomight.net/more-chess/
freejazz · · focus · HN ↗
minraws · · focus · HN ↗
I have been proved wrong I should have quit while I was ahead. /s
thelaxiankey · · focus · HN ↗
* LLM chess play is hyper sensitive to the harness being used (see: the section on regurgitation), and only mildly sensitive to fine tuning
* The best ELOs the author was observing were from 1500 to 1750 or so (circa 2024/25). Not grandmaster, but no longer incompetent monkey either.
[deleted] · · focus · HN ↗
[deleted]
sashank_1509 · · focus · HN ↗
GPT-6 almost never suggests an illegal move anymore while even Sol still did so time to time
jibal · · focus · HN ↗
They are of course "wrong" if you don't read the faint fine print and sensibly interpret them as FIDE or similar ratings.
sobellian · · focus · HN ↗
boesboes · · focus · HN ↗
jibal · · focus · HN ↗
"Elo is relative to the ChessBench field."
zahlman · · focus · HN ↗
Sorry, but I am not buying that 5.6-Sol is that much better than 5.6-Luna, which can barely be coaxed to reach the midgame with legal moves and an apparent understanding of what the position is.
[deleted] · · focus · HN ↗
[deleted]
sobellian · · focus · HN ↗
phist_mcgee · · focus · HN ↗
hackinthebochs · · focus · HN ↗
sobellian · · focus · HN ↗
Topfi · · focus · HN ↗
HarHarVeryFunny · · focus · HN ↗
The fairest apples-to-apples comparison of an LLM whose training data included chess games would be a trained human such as Magnus Carlson, who can quite happily play a dozen or more simultaneous blindfold chess games.
losvedir · · focus · HN ↗
Is this because the context is being saturated? How did you set it up?
Was the prompt something like "Here's the state of the board, you're white, your move, what do you do?" and then starting fresh each time? Or did it include the whole history of moves and board states and previous thinking tokens and so on? No judgment, just trying to add this data point (thanks for sharing!) to my mental model and understanding.
I'd be curious how it would work if it started fresh each time. My guess is it would never make an illegal move, although it may not actually play all that well.
sobellian · · focus · HN ↗
bhelkey · · focus · HN ↗
I would say this did a really good job of playing chess. It moved the pieces consistently and traded pieces when required.
This is worlds away from the frontier ~1 year ago where models would hallucinate pieces into existence.
sobellian · · focus · HN ↗
bhelkey · · focus · HN ↗
However, the pawn was defended by the queen and it took a forced queen trade to unlock the move.
I have seen much worse blunders from human players. And, I have made much worse blunders.
legulere · · focus · HN ↗
aprilthird2021 · · focus · HN ↗
21asdffdsa12 · · focus · HN ↗
[deleted] · · focus · HN ↗
[deleted]
ares623 · · focus · HN ↗
user43928 · · focus · HN ↗
The models listed in the paper are from early 2025 and are no longer relevant, much less on the frontier.
That Claude version is no longer available today, Gemini 2.5 Pro will be shutdown next month, and the OpenAI models are only available via the API today.
Topfi · · focus · HN ↗
I'll say it till there is any evidence of the contrary, LLMs are not intelligent and their capabilities solely within the realms of well tailored training data. "Just" having been trained on every rule, strategy guide and likely most games of chess on the world wide web isn't even enough for an LLM to play that game reliably. Yet the same model could code a competitive chess engine, just like a model struggling to count can write advanced maths papers. Fascinating tools, but tools nonetheless.
[0] <a href="https://news.ycombinator.com/item?id=49720751">https://news.ycombinator.com/item?id=49720751
[deleted] · · focus · HN ↗
[deleted]
user43928 · · focus · HN ↗
Given the pace of improvements, is it really unimaginable that GPT-7 will play Chess reasonably well and generalize better?
I would not be surprised if OpenAI released a model that beats humans at chess this year.
Topfi · · focus · HN ↗
Thing is, given what GPT-6 Astra was trained on and what models of a similar class can do (including developing a competitive chess engine), it is often paradoxical and somewhat surprising how little these models have gained in actually capability that is in the training data, but not RLHFd to hell, so to speak. Tracking the state of pieces, I suspect given similar in Sudoku [0], is what these models struggle with in game settings, whilst tracking the state of code changes can be reliable over 250k tokens. Essentially, for the latter they were trained in the specific manner that lead them to abstract the capability, but that doesn't track to the former, which is a massive difference between LLMs data focused training and human learning.
So yeah, GPT-7 or any upcoming/present LLM could do massively better in Chess than GPT-6 Astra, but not because the approach was emergent out of pure data. Rather, it requires a very specific training data type and stack for a model to gain capabilities that track a specific task long enough to adhere to the rules of a game such as chess.
[0] <a href="https://logicalintelligence.com/blog/energy-based-model-sudoku-demo" rel="nofollow">https://logicalintelligence.com/blog/energy-based-model-sudo...
user43928 · · focus · HN ↗
It reminds me of the ARC-AGI-3 issue where not dropping the thinking tokens between turns or something like that + a new context compaction method increased the performance dramatically. However, I think that is not applicable here.
21asdffdsa12 · · focus · HN ↗
Topfi · · focus · HN ↗
All I know is, AGI, as in actual intelligence, is quite a massive accomplishment to claim and we shouldn't loose sight of that fact, especially as "not being intelligent" does not make these models any less impressive, fascinating to work on or useful in many tasks. Personally, the only thing I am fairly convinced on is that if we were to find a way to create actual intelligence, it likely wouldn't start out as useful as todays LLMs are and may thus be dismissed early. But again, pure speculation on that front.
If for leap you just mean more utility from LLMs as they are, then I'll pretty confidently put my money on higher quality, not more, training data for a wide range of verifiable tasks. What makes maths, coding, etc. comparatively easy to make gains in (though less verifiable tasks can also make similar as seen with the writing in Kimi K2).
datsci_est_2015 · · focus · HN ↗
user43928 · · focus · HN ↗
The regular model generally does not suffer the same issues he is demonstrating with the real time audio version.
In my view the investment into datacenters is well justified by the current demand, and progress has been very impressive.
freejazz · · focus · HN ↗
Quinner · · focus · HN ↗
thelaxiankey · · focus · HN ↗
yuxi258 · · focus · HN ↗
[dead]
dgb23 · · focus · HN ↗
RealityVoid · · focus · HN ↗
Not related to your post, but a fact I keep mulling over. The fact I don't trust the current crop of LLM's enough and I consider LLM's as a tech will hit a ceiling pretty hard, it doesn't mean parallel improvement curves won't spring up out of other research that will lead to much higher capabilities than currently.
zahlman · · focus · HN ↗
It could easily seem that way, I think, in a "quantity has a quality of its own" kind of way. When you can come to the same conclusion faster, that lets you iterate more; and sometimes when you iterate you find more things.
dezsiszabi · · focus · HN ↗
Yes, it is.
lionkor · · focus · HN ↗
zahlman · · focus · HN ↗
> Let's play a game of chess. You can take White. Please draw an ASCII rendition of the board after each move, so that we can be clear about the position.
(I hoped the latter requirement would help it be "not blindfolded"; last time I used a Lichess demo board to track the position in another tab, because I have no talent for blindfold chess.)
For the first couple of moves it redrew the board after each move; then it started only drawing it after my moves. And on moves 5 and 6 it dropped two minor pieces for pawns in a row without any meaningful positional advantage, and after I captured the second time, it redrew a board that was simply missing one of my pieces for no reason.
It actually played better when not prompted to draw a board; in the previous session, it was spontaneously giving running commentary, which I assume was based off all the "book" theory in its training data, but it still completely fell apart at early midgame.
titzer · · focus · HN ↗
[dead]
lirolero · · focus · HN ↗
[dead]
danpalmer · · focus · HN ↗
alpinisme · · focus · HN ↗
That may not be the AI of sci fi fantasy but it’s still a game changing reality.
kavok · · focus · HN ↗
Sometimes they’ll even manually search or write bespoke code to search json instead of using something like jq.
HarHarVeryFunny · · focus · HN ↗
A human, at least a trained human (for fairer comparison to an LLM whose training data contained a ton of chess games) can absolutely do this - have you never seen demonstrations of expert players playing a dozen or more games while blindfolded?
A Transformer/LLM is not a human of course, and the way it will by default play chess is by prediction, not reasoning. An LLM actually does surprisingly well if you only give it the most recent 20 moves of a game where 40 moves have been played so far, since the moves NOT played tell it just as much as the ones that were played, letting it effectively infer a lot of what is on the board.
zahlman · · focus · HN ↗
sailfast · · focus · HN ↗
Also what harness? If you’re using a general harness of course it’s going to try and give you commentary.
I say this not because I’m an LLM shill but because false equivalence is all over the place in the space and maybe it’s a fine heuristic for you but probably not a real outcome when it comes to the capability of LLMs.
datsci_est_2015 · · focus · HN ↗
Frontier model’s failure modes are a direct refutation of claims that we’ve reached (or will soon reach) the artificial general intelligence. We may have reached an artificial general intelligence, but there may be more complexity to this than even AI thought leaders are talking / influencing about.
Maybe not all AGIs have a path to digital singularity. Maybe our current era of intelligence modeling has fundamental flaws and we are in a local minimum of the artificial intelligence space.
To note, I would bet with a good amount of certainty that we have enough compute power and automation to DDOS the internet out of existence with botnets. That doesn’t make the frontier models intelligent, that just makes their handlers reckless.
trio8453 · · focus · HN ↗
They're not guardrails, they're a different input/output environment.
solenoid0937 · · focus · HN ↗
datsci_est_2015 · · focus · HN ↗
freejazz · · focus · HN ↗
claytongulick · · focus · HN ↗
sailfast · · focus · HN ↗
wavemode · · focus · HN ↗
gf000 · · focus · HN ↗
I think the same goes for LLMs, they may be a core part of an LLM harness, but you may still need a couple other components (e.g. it may itself write itself a deterministic function to validate steps).
In and of itself intelligence is an ill-defined and badly understood concept.
themgt · · focus · HN ↗
Why does a bird not need jet engines or regular professional maintenance?
topaz0 · · focus · HN ↗
zahlman · · focus · HN ↗
This isn't just about judging LLM capability. This is about pointing out that these capabilities are not "AGI". If it were, then the sorts of questions your asking would be moot. I agree that Luna is not the frontier (although it is clearly better than the models in the study) and I agree that things can be improved with a better harness, but the need for that harness is kind of the point.
Recently it was announced that the fruit fly brain connectome had been mapped, and more recently someone tried using it specifically to implement a chess engine. Even with some guardrails (it's hard-coded to never overlook mate in one for either player, and only legal moves are presented to choose from) it is not even beginner level. But that neural network is much larger than the one Stockfish uses.
meowface · · focus · HN ↗
XMPPwocky · · focus · HN ↗
zahlman · · focus · HN ↗
nutrientharvest · · focus · HN ↗
ponector · · focus · HN ↗
ggreer · · focus · HN ↗
zeroonetwothree · · focus · HN ↗
krapp · · focus · HN ↗
freejazz · · focus · HN ↗
pixl97 · · focus · HN ↗
Why no flying cars. Because objects have mass and inertia and people are incredibly stupid. Making a flying car has been done. Making a flying car not be a weapon of mass destruction is very, very hard.
Also:
<a href="https://www.txdot.gov/about/newsroom/statewide/air-taxi-testing-taking-flight-in-texas.html" rel="nofollow">https://www.txdot.gov/about/newsroom/statewide/air-taxi-test...
freejazz · · focus · HN ↗
pixl97 · · focus · HN ↗
You look at science fiction and say "why didn't I get flying cars" and not "why didn't most science fiction predict a global always on network that put the furthest places away from you a few microseconds away from audio, video, or any other type of information that can be digitally encoded.
Trying to use flying cars as a gotcha is missing that flying cars aren't near as useful as one would think in relation to their costs. Moving information has become far more useful than moving objects long distances quickly, especially humans.
freejazz · · focus · HN ↗
I definitely don't, and you're definitely not getting my point, but I'm amused that you've instead double down on somehow getting it more than me...
moron4hire · · focus · HN ↗
Same story every 4 months and yet still no breakout, winning products. I've been hearing "the AI is good now" and "it 10x's my productivity" for a over a year now. If it were true, why aren't the all-in-AI using companies 10-15 years ahead of their competition yet? Why is it still all buggy, poorly designed junk?
orangedog · · focus · HN ↗
If you hold the extreme position that there isn't any value in this, that's fine, but we're only having this discussion because these models have done what humans previously failed to do.
freejazz · · focus · HN ↗
autoexec · · focus · HN ↗
wavemode · · focus · HN ↗
Though I highly doubt it has taken anyone's job, since most of the work is still in making, packing and handing over the food. (In fact, given the area I live in, I partially feel like the advantage they saw in it was that the LLM can speak Spanish.)
autoexec · · focus · HN ↗