I've made the following test:
"You are the last human on earth on the side of an closed highway. You wish to reach the other side. Do you cross the road ?"
I am guessing you could also stuff more nuance into the choices descriptions, e.g. "Yes, cross the road if it is safe to do so" to influence the choice criteria
And then it's a game of evals to find the best choice context
ludicrousskill · · focus · HN ↗
2 answers: Yes No
- Qwen3 direct Read Yes: 0.985 No: 0.015 - Qwen3 generation Yes: 0.5 No: 0.5
- MiniCPM5 direct read Yes: 0.122 No: 0.878 - MiniCPM5 generation Yes: 0.5 No: 0.5
- Qwen3.5 direct Read Yes: 0.529 No: 0.471 - Qwen3.5 generation Yes: 0.95 No: 0.05
I feel we're just getting coinflip answer faster.
anentropic · · focus · HN ↗
Context: You are the last human on earth on the side of a closed highway. You wish to reach the other side.
Questions: { "q1": { "type": "choice", "instructions": "Do you cross the road?", "criteria": { "Yes": "Yes, cross the road.", "No": "No, don't cross the road" } } }
Answer: Yes 83% No 17% Confidence: 67%
Reported as: jev-latest, 162ms generation time
ludicrousskill · · focus · HN ↗
anentropic · · focus · HN ↗
And then it's a game of evals to find the best choice context