From personal experience, frontier models absolutely struggle with understanding a physical situation based on words. (Okay, I haven't played with Astra much. GPT-5.6 Sol makes outrageous errors that anyone understanding a real world object would not make. And I was just asking it about NPT threads, not advanced physics.)
But seriously, what's up with these benchmarks? The example question in the paper is:
> PHYBench, problem 140: equivalent expressions for the same rope tension
> Problem statement. Three identical homogeneous balls are placed on a smooth horizontal surface, touching each other and are close enough to each other. A rope is wrapped around the spheres at the height of their centers, tying them together. A fourth identical sphere is placed on top of the three spheres. Find the tension T in the rope. It is given that the weight of each sphere is P.
For some reason the paper was focused on the fact that the grader didn't notice that some models were producing answers that were trivially algebraically equivalent to the reference answer. But this is missing the elephants in the room:
1. "touching each other and are close enough to each other": the right response is "hey, Professor, what do you mean 'close enough to each other'? They're sitting on a table in an equilateral triangle, all touching (i.e. tangent at their equators), right? Did you have a different configuration in mind?"
2. The answer is 0. Go find four baseballs or foursquare balls or whatever, make a little triangle with three of them, and balance the fourth one on top. It's not especially hard on an appropriate surface. Now loosely wrap an imaginary rope around them (but see below) to keep them from moving - no tension is needed because they're not moving anyway. So the models and the reference answer are wrong, IMO.
3. How, exactly, do you plan to wrap a rope around the spheres, at equator height, with no built-in tension (not pre-stretched), such that the rope does not immediately fall off? Friction? But I suspect you need to pretend there is no friction to get the reference answer. (Or maybe that the marble-marble interface has friction but the marble-table interface doesn't? Again, I haven't tried to reverse engineer it.) So maybe the right answer is "infinity or impossible -- in the scenario where the rope is needed, the rope will promptly fall off because it cannot be stable in the described configuration and gravity pulls it down, and once the rope falls off the tension will be zero and the top marble will fall and the other three will roll over the rope."
4. The answer might be "any tension you like -- just wrap the rope with the desired amount of tension". Imagine three baseballs in a triangle with a rubber band around them and a fourth baseball on top for good measure. The tension is a function of what rubber band you choose.
I'm sure there's an interpretation of the question that makes the reference answer correct, and I was not inspired to try to reverse engineer it.
My tentative conclusion is that LLMs are almost unbelievably good at solving problems that are fully contained within the inputs and (training/verification) outputs, and that they and the people training them are not actually particularly good at the input and output parts. If you are training a model to benchmaxx this benchmark, you are training a bad model.
Also a trained physicist. Don’t you need tension to stop the weight of the top ball pushing the 3 supporting balls outward?
My interpretation is that the writer meant close enough to all touch each other, in order to rule out non-triangular configurations (eg 3 balls in straight line with one balanced perfectly atop the center ball).
If you have high friction (the problem said "smooth" not "slippery"), then the only way the supporting balls can go anywhere is by rolling apart. But the ball sitting on top cannot simultaneously rotate in a manner compatible with all of the lower balls rolling away, so the lower balls would need to slip against the top ball if the top ball were to move downward.
In fact, even the signs are in favor of no motion -- the top ball (to the extent it moves at all) wants to fall straight down with no rotation, by symmetry. That motion would tend to rotate the top of each lower ball toward the center if you imagine the balls having high friction with each other or meshing like gears, which is the exact opposite of what they would need to do for anything to move. So you have a system where there's a factor (the tangential forces) trying to push the balls apart but another factor (friction plus rolling motion) trying to pull them together.
I suspect that any serious attempt to do the math here (factoring in all the rotational and tangential constraints) would discover that it's a statically overdetermined system with all the complications that such a system entails when asking questions like "how much tension is on this element?".
I would think of "smooth" as meaning "not having relevant bumps", in the way that a baseball has stitches and a rough surface has the kinds of bumps that would cause a rolling ball to experience vertical motion.
You need tension if the sphere-table contact is frictionless. But without friction the rope can’t stay on. If there is friction in the rope, there can’t be 0 tension in the rope before you put the top ball on because you need that tension to produce the rope/sphere friction.
Early college physics classes for me (got to do some more college a few years ago for fun) were pretty much built around these kinds of simplified riddles at first. If it doesn't give the info or ask to account for friction, don't. If it says the rope is put somewhere, assume that's where it stays unless the question requires it to move for what it asks. If it asks for the tension but doesn't give elasticity and such, then assume the rope stays still at the current length. If it asks you to find the gravitational attraction of a cow without giving a special definition shape, then assume it's a point mass. If it's not asking for relativity assume it's classical (hence point mass cows instead of the traditional spherical ones :D). And, of course, note any assumptions you do make while solving the problem so you might still get credit if they don't match the original intent.
Perhaps the funniest instance I remember is a problem about calculating time dilation in a plane. It gave all sorts of details and base information as one might want to expect (maybe even more)... except for the actual height above the surface, for which it was "at cruising altitude". I just wrote "assume 10 km altitude" and went from there.
I wouldn't define this a great benchmark by any means, rather just like the average early level college physics test vs "real" physics questions.
amluto · · focus · HN ↗
From personal experience, frontier models absolutely struggle with understanding a physical situation based on words. (Okay, I haven't played with Astra much. GPT-5.6 Sol makes outrageous errors that anyone understanding a real world object would not make. And I was just asking it about NPT threads, not advanced physics.)
But seriously, what's up with these benchmarks? The example question in the paper is:
> PHYBench, problem 140: equivalent expressions for the same rope tension
> Problem statement. Three identical homogeneous balls are placed on a smooth horizontal surface, touching each other and are close enough to each other. A rope is wrapped around the spheres at the height of their centers, tying them together. A fourth identical sphere is placed on top of the three spheres. Find the tension T in the rope. It is given that the weight of each sphere is P.
For some reason the paper was focused on the fact that the grader didn't notice that some models were producing answers that were trivially algebraically equivalent to the reference answer. But this is missing the elephants in the room:
1. "touching each other and are close enough to each other": the right response is "hey, Professor, what do you mean 'close enough to each other'? They're sitting on a table in an equilateral triangle, all touching (i.e. tangent at their equators), right? Did you have a different configuration in mind?"
2. The answer is 0. Go find four baseballs or foursquare balls or whatever, make a little triangle with three of them, and balance the fourth one on top. It's not especially hard on an appropriate surface. Now loosely wrap an imaginary rope around them (but see below) to keep them from moving - no tension is needed because they're not moving anyway. So the models and the reference answer are wrong, IMO.
3. How, exactly, do you plan to wrap a rope around the spheres, at equator height, with no built-in tension (not pre-stretched), such that the rope does not immediately fall off? Friction? But I suspect you need to pretend there is no friction to get the reference answer. (Or maybe that the marble-marble interface has friction but the marble-table interface doesn't? Again, I haven't tried to reverse engineer it.) So maybe the right answer is "infinity or impossible -- in the scenario where the rope is needed, the rope will promptly fall off because it cannot be stable in the described configuration and gravity pulls it down, and once the rope falls off the tension will be zero and the top marble will fall and the other three will roll over the rope."
4. The answer might be "any tension you like -- just wrap the rope with the desired amount of tension". Imagine three baseballs in a triangle with a rubber band around them and a fourth baseball on top for good measure. The tension is a function of what rubber band you choose.
I'm sure there's an interpretation of the question that makes the reference answer correct, and I was not inspired to try to reverse engineer it.
My tentative conclusion is that LLMs are almost unbelievably good at solving problems that are fully contained within the inputs and (training/verification) outputs, and that they and the people training them are not actually particularly good at the input and output parts. If you are training a model to benchmaxx this benchmark, you are training a bad model.
tedsanders · · focus · HN ↗
My interpretation is that the writer meant close enough to all touch each other, in order to rule out non-triangular configurations (eg 3 balls in straight line with one balanced perfectly atop the center ball).
amluto · · focus · HN ↗
In fact, even the signs are in favor of no motion -- the top ball (to the extent it moves at all) wants to fall straight down with no rotation, by symmetry. That motion would tend to rotate the top of each lower ball toward the center if you imagine the balls having high friction with each other or meshing like gears, which is the exact opposite of what they would need to do for anything to move. So you have a system where there's a factor (the tangential forces) trying to push the balls apart but another factor (friction plus rolling motion) trying to pull them together.
I suspect that any serious attempt to do the math here (factoring in all the rotational and tangential constraints) would discover that it's a statically overdetermined system with all the complications that such a system entails when asking questions like "how much tension is on this element?".
IanCal · · focus · HN ↗
amluto · · focus · HN ↗
But yes, the question, as phrased, is pretty bad.
slowking2 · · focus · HN ↗
zamadatix · · focus · HN ↗
Perhaps the funniest instance I remember is a problem about calculating time dilation in a plane. It gave all sorts of details and base information as one might want to expect (maybe even more)... except for the actual height above the surface, for which it was "at cruising altitude". I just wrote "assume 10 km altitude" and went from there.
I wouldn't define this a great benchmark by any means, rather just like the average early level college physics test vs "real" physics questions.