I tested Jev with a fair die 400 times without telling it the die result. The true probability of face 1 is 1/6, but Jev always chose face 1 and the probability it returned was about 83%. I also tested with a fair coin 200 times and got 0.92 probability.
I did several tests and I think Jev is good at problems with a correct answer but weak at problems about actual probabilities whose answers can't be known at all.
Write-up: "Jev Does Not Play Dice"
<a href="https://kantahayashiai.github.io/posts/jev-does-not-play-dice/" rel="nofollow">https://kantahayashiai.github.io/posts/jev-does-not-play-dic...
> The true probability of face 1 is 1/6, but Jev always chose face 1 and the probability it returned was about 83%
I think this is a misunderstanding of what that 83% probability means. You think you're asking Jev to assign a probability to a dice roll, and if so then obviously it should pick each answer 1/6 of the time.
However, what Jev is doing is assigning a probability to the answer it should choose. And it's choosing 1 with 83% probability, as the correct answer to give, when there's no additional knowledge available.
1 is a perfectly valid answer here. In the absence of other data there's no more (nor less) valid answer than that. and I assume that just like LLMs, the token prediction must assign higher probability to one outcome.
If you keep feeding the previous answers back in (if you can do that with Jev) I assume it would start giving other numbers because it's no longer operating from zero info. Maybe that would even reach ~1/6 after enough rounds, although maybe not.
As someone who hasn’t used Jev, this seems reasonable in the sense that “probability that I’ve given the best answer” seems like a useful thing to provide.
Anyway, if it is “probability that I gave the best answer” why 83%? I mean, the question is a sort of a textbook-style probability question in the sense that everything is well defined. Shouldn’t it be basically 100%?
kantahayashi · · focus · HN ↗
I did several tests and I think Jev is good at problems with a correct answer but weak at problems about actual probabilities whose answers can't be known at all.
Write-up: "Jev Does Not Play Dice" <a href="https://kantahayashiai.github.io/posts/jev-does-not-play-dice/" rel="nofollow">https://kantahayashiai.github.io/posts/jev-does-not-play-dic...
esperent · · focus · HN ↗
I think this is a misunderstanding of what that 83% probability means. You think you're asking Jev to assign a probability to a dice roll, and if so then obviously it should pick each answer 1/6 of the time.
However, what Jev is doing is assigning a probability to the answer it should choose. And it's choosing 1 with 83% probability, as the correct answer to give, when there's no additional knowledge available.
1 is a perfectly valid answer here. In the absence of other data there's no more (nor less) valid answer than that. and I assume that just like LLMs, the token prediction must assign higher probability to one outcome.
If you keep feeding the previous answers back in (if you can do that with Jev) I assume it would start giving other numbers because it's no longer operating from zero info. Maybe that would even reach ~1/6 after enough rounds, although maybe not.
bee_rider · · focus · HN ↗
Anyway, if it is “probability that I gave the best answer” why 83%? I mean, the question is a sort of a textbook-style probability question in the sense that everything is well defined. Shouldn’t it be basically 100%?