I tested Jev with a fair die 400 times without telling it the die result. The true probability of face 1 is 1/6, but Jev always chose face 1 and the probability it returned was about 83%. I also tested with a fair coin 200 times and got 0.92 probability.
I did several tests and I think Jev is good at problems with a correct answer but weak at problems about actual probabilities whose answers can't be known at all.
Write-up: "Jev Does Not Play Dice"
<a href="https://kantahayashiai.github.io/posts/jev-does-not-play-dice/" rel="nofollow">https://kantahayashiai.github.io/posts/jev-does-not-play-dic...
Humans give way more random answers than LLMs to questions like "give me a random number between 1-100" (when not giving the LLM any tool calls).
kantahayashi · · focus · HN ↗
I did several tests and I think Jev is good at problems with a correct answer but weak at problems about actual probabilities whose answers can't be known at all.
Write-up: "Jev Does Not Play Dice" <a href="https://kantahayashiai.github.io/posts/jev-does-not-play-dice/" rel="nofollow">https://kantahayashiai.github.io/posts/jev-does-not-play-dic...
scotty79 · · focus · HN ↗
Humans also don't give a perfect 1/n probability when asked for a random number.
formerly_proven · · focus · HN ↗
tomrod · · focus · HN ↗
<a href="https://www.youtube.com/watch?v=d6iQrh2TK98" rel="nofollow">https://www.youtube.com/watch?v=d6iQrh2TK98