Thread

1. qt31415926 · 1789586348 · focus · HN ↗

   Article: "How Good Are Frontier Models at Physics? Expert Re-Grading Reveals Broken Evaluations and Near-Saturation of Leading Benchmarks"

   John Sous from Yale posted a fairly solid study on how nearly all the physics benchmarks are broken, as they consistently mark correct answers as incorrect.

   When hand grading instead, they found out that the models have actually already saturated the benchmarks which is a little bit scary.

   1. fsh · 1789588970 · focus · HN ↗

      I would be very surprised if any of the frontier models wasn't trained on all public physics benchmarks. Training data providers have been hiring people for exactly this task.

      1. letmevoteplease · 1789591417 · focus · HN ↗

         This study appears to be evidence against that: the model failed the benchmark but arrived at the correct answer.

         1. fsh · 1789591802 · focus · HN ↗

            Half of the benchmarks don't have published answers. That's why training data providers have been hiring physicists to solve them.

            1. [deleted] · 1789592792 · focus · HN ↗

               [deleted]

            2. red75prime · 1789602144 · focus · HN ↗

               This is unconventional benchmaxxing then, when they decrease the benchmark scores to allow models to generalize on correct solutions.

Unofficial Hacker News client; not affiliated with Y Combinator.