Hasn't there been a lot talk about Astra's opaque reasoning capabilities (being able to think through complex questions without using a chain of thought)?
Given that, can't you just replicate Jev by telling Astra "here is the question, you must make a multiple choice decision / output a score between 1-10, please answer directly in a single word, no reasoning allowed"?
(Edit: Ok, Jev is much cheaper in input tokens so these two aren't directly comparable at all)
2001zhaozhao · · focus · HN ↗
Given that, can't you just replicate Jev by telling Astra "here is the question, you must make a multiple choice decision / output a score between 1-10, please answer directly in a single word, no reasoning allowed"?
(Edit: Ok, Jev is much cheaper in input tokens so these two aren't directly comparable at all)
CompleteSkeptic · · focus · HN ↗
we actually use astra (and fable) in this way for our evals: evals.typesafe.ai
someone on the team cooked hard on that and it shows example traces comparing our model to opus/sol