I wonder if that means that SpaceX evals show that they consider astra better than fable or that they hate Sam&co so much they don't want to show their stuff.
Its because of this. You can't use Astra in Cursor, and cursorbench uses cursor as the harness. They can't actually benchmark it using their harness hence why its not included.
This explains why. Mentioned in another comment, but cursorbench explicitly tests with Cursor as the harness, and OpenAI doesn't allow them to use Astra in Cursor.
That's not accurate. OpenAI doesn't allow Grok to provide Astra to Cursor customers anymore, but it doesn't ban anyone from using Astra via alternative harnesses.
If Cursor wanted to include Astra in CursorBench nothing would stop them, they could easily have spent half an hour vibecoding in OpenAI API key support - if it hadn't been convenient to neglect to do that.
Even if they could do that (workaround to include Astra in CursorBench), that has no practical consequences for Cursor users and that's what I as a Cursor user (what I use for dev, though I use ChatGPT for non-dev stuff) care about.
The point of Cursor Bench is to show how models perform in Cursor. If 99% of their users won't be able to access a model unless they go out of their way to include setup an API key for it (which would be insanely expensive with Astra), why would they include it in the benchmark?
If the benchmark is "what's the best model to use in your Cursor subscription", why would they do that? OpenAI knew what the effects of their decision were. Hard for me to have sympathy for either party here, honestly, and I say that as someone who is a customer of both.
Cursor never added Astra to its consumer subscription plans. And it's likely exactly because of this announcement. Why would they add support for a model they would have to remove shortly after?
Deceptive? An extremely quick google search would answer your question. OpenAI pulled out of Cursor before they released Astra so it never got that benchmark.
kristofferR · · focus · HN ↗
Iolaum · · focus · HN ↗
babelfish · · focus · HN ↗
Jcampuzano2 · · focus · HN ↗
Its because of this. You can't use Astra in Cursor, and cursorbench uses cursor as the harness. They can't actually benchmark it using their harness hence why its not included.
babelfish · · focus · HN ↗
Jcampuzano2 · · focus · HN ↗
ryeguy · · focus · HN ↗
bluecalm · · focus · HN ↗
<a href="https://x.com/elonmusk/status/2102082011233931762?s=20" rel="nofollow">https://x.com/elonmusk/status/2102082011233931762?s=20
so it's likely about usage in Cursor specifically.
DavCreator · · focus · HN ↗
Jcampuzano2 · · focus · HN ↗
This explains why. Mentioned in another comment, but cursorbench explicitly tests with Cursor as the harness, and OpenAI doesn't allow them to use Astra in Cursor.
kristofferR · · focus · HN ↗
If Cursor wanted to include Astra in CursorBench nothing would stop them, they could easily have spent half an hour vibecoding in OpenAI API key support - if it hadn't been convenient to neglect to do that.
andsoitis · · focus · HN ↗
kristofferR · · focus · HN ↗
Jcampuzano2 · · focus · HN ↗
mh- · · focus · HN ↗
user43928 · · focus · HN ↗
That said, I don't expect them to benchmark Astra in their Cursor harness given the situation.
Jcampuzano2 · · focus · HN ↗
oh_no · · focus · HN ↗
scottyah · · focus · HN ↗
kristofferR · · focus · HN ↗
scottyah · · focus · HN ↗