I tested Jev with a fair die 400 times without telling it the die result. The true probability of face 1 is 1/6, but Jev always chose face 1 and the probability it returned was about 83%. I also tested with a fair coin 200 times and got 0.92 probability.
I did several tests and I think Jev is good at problems with a correct answer but weak at problems about actual probabilities whose answers can't be known at all.
Write-up: "Jev Does Not Play Dice"
<a href="https://kantahayashiai.github.io/posts/jev-does-not-play-dice/" rel="nofollow">https://kantahayashiai.github.io/posts/jev-does-not-play-dic...
In the early Gemini 2 days (don't remember which version exactly) I had Gemini running as a voice assistant in my kitchen, and asked it to flip a coin and tell me if it was heads or tails. It responded with "heads". I was curious if it was actually doing something to simulate randomness, so I asked a few more times and saw a pattern: "tails", "heads", "tails", "heads"...
It continued alternating between the two until I got bored (around a dozen turns).
Unless your specific test is baked into its training, real probabilities require math and rough approximation at a minimum needs reasoning to sanity-check. Jev does neither. This isn't a new problem or anything unique to Jev.
kantahayashi · · focus · HN ↗
I did several tests and I think Jev is good at problems with a correct answer but weak at problems about actual probabilities whose answers can't be known at all.
Write-up: "Jev Does Not Play Dice" <a href="https://kantahayashiai.github.io/posts/jev-does-not-play-dice/" rel="nofollow">https://kantahayashiai.github.io/posts/jev-does-not-play-dic...
drtz · · focus · HN ↗
It continued alternating between the two until I got bored (around a dozen turns).
Unless your specific test is baked into its training, real probabilities require math and rough approximation at a minimum needs reasoning to sanity-check. Jev does neither. This isn't a new problem or anything unique to Jev.
DuperPower · · focus · HN ↗