Looking at the logprobs on tokens works for the local models, but not on the frontier ones. It's been more or less broken since GPT-4o for example. I wrote about it two years ago: <a href="https://medium.com/data-science/9-11-or-9-9-which-one-is-higher-6efbdbd6a025" rel="nofollow">https://medium.com/data-science/9-11-or-9-9-which-one-is-hig.... Also, I've done some work in estimating confidence and on rubric evals using the same method, and you actually get better correlation to "real confidence" by just getting the LLM to say it.
Interesting. Have you repeated these experiments with recent models? I'm thinking frontier models APIs have tools/MCPs for math stuff but curious about recent Qwen models, etc.
Haven't tried recent frontier ones, Astra for example doesn't support logprobs emission on the API, and Sol and Luna supposedly support it with reasoning disabled. Haven't tried local models like Qwen 3.8 27b (I'm actually exploring their thinking trace, it's a lot of fun)
armcat · · focus · HN ↗
hununu · · focus · HN ↗
armcat · · focus · HN ↗