Cool approach, i think less latency and cost is the way to go.
Here's how this would have likely been made.
- Tiny transformer or equivalent model (maybe a few bn or so?), explaining latency and cost
- Questions are sent in parallel to multiple copies of it (I'm sure they're edge located)
- The model is post-trained for calibration in a wide variety of data (the recipe is relatively simple, and likely targeted on distillation of logprobs / confidence of a bigger model)
Notice how cost is ONLY for input tokens as output is merely numbers (few tokens) because input could be huge (questions and options).
At 0.042-per-million price they have, Astra estimates the model to be 3bn parameters.
One could replicate this by post training Qwen 3.5 2Bn. I expect people to do so soon!
paraschopra · · focus · HN ↗
Here's how this would have likely been made.
- Tiny transformer or equivalent model (maybe a few bn or so?), explaining latency and cost
- Questions are sent in parallel to multiple copies of it (I'm sure they're edge located)
- The model is post-trained for calibration in a wide variety of data (the recipe is relatively simple, and likely targeted on distillation of logprobs / confidence of a bigger model)
Notice how cost is ONLY for input tokens as output is merely numbers (few tokens) because input could be huge (questions and options).
At 0.042-per-million price they have, Astra estimates the model to be 3bn parameters.
One could replicate this by post training Qwen 3.5 2Bn. I expect people to do so soon!
niutech · · focus · HN ↗