Thread

1. breckenedge · 1790098407 · focus · HN ↗

   Do these evaluations get re run a few weeks after launch? I started doing that yesterday for our internal dataset and found Sol’s performance had regressed to be equal to Luna’s. Granted this was one run, but something I’m becoming more concerned about, the model providers want to quickly prove they’re the best, people switch to them, then they pull the rug.

   1. tedsanders · 1790116783 · focus · HN ↗

      Can you share more information on your methodology?

      GPT-5.6 Sol's performance in the API should not change over time. If it has, that's a severe bug and we'll look into it.

      We do sometimes tweak ChatGPT settings (e.g., tools, system prompts, efforts) over time, but we never play games to juice evals at launch times. You should always get what's advertised.

      (I work at OpenAI.)

      1. sunaurus · 1790160479 · focus · HN ↗

         Do you have any hypothesis for why this experience is so consistently reported by users (anecdotally)? Seemingly across all providers.

         1. tedsanders · 1790177307 · focus · HN ↗

            My best guess is random variation and rising expectations. I once saw someone do in depth manual testing of 3 different models and write up a whole report on their perceived strengths and weaknesses… and then discover that all 3 models were identical. Very good reminder of why the scientific method is important.

            Serial testing over time is much less reliable than side by side testing, and even when I do side by side testing, I try to look at multiple attempts per prompt. Seeing multiple per prompt helps me realize how much intrinsic variation there is. My brain always wants to see patterns even when there isn’t enough data to prove them.

Unofficial Hacker News client; not affiliated with Y Combinator.