Thread

GPT 6.1 Sol: Near-Astra intelligence for a fifth of the price

1066 points · 953 comments · crorella

minimaxir · 1790702170 · focus · HN ↗
> Cached input costs just $0.10 per million tokens—95% less than standard input pricing and 50% less than GPT‑6 Sol’s cached input pricing
This is the actual big announcement. 50% cheaper cache than GPT-6 Sol will get you far more mileage on Codex.
1. TuxSH · 1790702497 · focus · HN ↗
  Exactly half as expensive as Opus 5.5 in every API pricing metric
  1. bigwheels · 1790703039 · focus · HN ↗
    And half as good. I didn't have great experiences with Anthropic models in the past, but Opus 5.5 seems to have turned a major corner. It is churning through tasks significantly more quickly and efficiently.
    Suggest trying it out yourself: Ask for something difficult from GPT-6 Sol and Opus 5.5 and watch what each one does. The difference is stark.
    Edit: Defining "difficult" as a complex coding or systems task (or even series of them in a single prompt).
    Infinity315 · 1790703186 · focus · HN ↗
    I'm not an OpenAI simp, but how anyone can have any opinion on the performance of these models in less than a day - let alone a few hours - is beyond me.
    toasty228 · 1790703298 · focus · HN ↗
    Try it, it's that good compared to openai current offering.
    I get better results and usage our of my $20 claude sub than my $100 openai sub... it's that ridiculous
    copperx · 1790703714 · focus · HN ↗
    [delayed]
    AndrewKemendo · 1790703338 · focus · HN ↗
    Only takes 5-10 minutes to test your favorite one shot comparison prompt.
    squidbeak · 1790703980 · focus · HN ↗
    If 5-10 minutes is enough, you need a more ambitious one-shot goal.
    edgyquant · 1790704201 · focus · HN ↗
    Can you give an example? For me I find that one shot prompts are pretty good it’s only when working with large codebases and complex, multi prompt workflows, that I find the real limitations of models
    AndrewKemendo · 1790708580 · focus · HN ↗
    Yeah the whole Pelican riding the bike is the best obvious one
    [deleted] · 1790703344 · focus · HN ↗
    [deleted]
    colinhb · 1790704025 · focus · HN ↗
    Yeah totally agree, people keep jumping in w/ strong views hours after release, eg: <a href="https://news.ycombinator.com/item?id=49045430">https://news.ycombinator.com/item?id=49045430
    rspeele · 1790704573 · focus · HN ↗
    While I have no experience comparing this brand-new model, OpenAI themselves call it "near-Astra" intelligence. I set Astra and Opus 5.5 independently working on the same large research/coding task in an experimental project (doing NURBS surface modeling stuff). They had the same starting repo state, same task packet, same test suite to try to meet. I have the $100 plan in both.
    Astra used 215% of a week's budget (I burned 2 free resets) and took 13 hours. Opus used 20% of a week's budget and took 20 hours. Both were asked to use lesser sub-agents for implementation grunt work at their discretion (Luna, Sonnet) as long as they manage and review the output.
    The timing comparison is not that interesting because the wall-clock speed mostly reflects how often they ran the (large, slow) test suite, not their coding speed. Although in the past my gut feeling is that OpenAI models do generally respond faster.
    The quality of their implementation was more interesting. There turned out to be a bug in one of the unit tests the agents were trying to pass. Opus interpreted the natural-language requirements from the task packet, found the test bug, and fixed it. Astra tried hard to solve the problem without altering the test suite. In practical terms Opus got much, much farther into a useful implementation. Astra was still stubbing out and faking critical parts of the implementation (B-splines) and since it ultimately couldn't pass the full test suite, finally gave up on its implementation. Astra wrote some useful tooling in the process of its efforts which I ended up integrating into Opus's version of the code, but otherwise its approach was behind.
    Now, this is just one comparison in one domain, and arguably Astra's strict adherence to the tests as-given is a good thing. But Opus wasn't merely loosening the rules / moving the goalposts to pass, it spotted an actual bug, and was more successful at doing what I actually wanted. And the cost difference was Astra-nomical.
    Out of curiosity for an interpretation free from my personal bias, I gave Astra a hint from Opus and permission to change the test in question, which it did, and got a bit farther, but still ultimately didn't produce a working implementation (to be fair, Opus's was not completely working either, but was closer). I then fired up fresh agents to review the two repos. Predictably, an Opus agent thought the Opus-written repo was the better basis to build on, and an Astra agent thought the Astra-written repo was the one to keep. They were not explicitly told which was which nor did the commit trailers say, but I assume they can tell. However, after doing this twice each, I saved the 4 review reports into another folder and did yet another meta-review of the 4 reports, so each would see the arguments and critiques both directions. In this meta-review both Astra and Opus converged on preferring the Opus implementation.
    agar · 1790707976 · focus · HN ↗
    This was a very interesting, informative, and well-written comment (and experiment). Thank you.
    this_user · 1790719544 · focus · HN ↗
    Astra doesn't just burn token at an insane rate, it is also strangely high maintenance when using it. Occasionally, you have to keep prodding it to keep working. Then at other times, it will disappear down some rabbit hole, trying to resolve increasingly hypothetical issues. It feels like you constantly have to keep it on track, while Opus is just churning through tasks.
    rrvsh · 1790728240 · focus · HN ↗
    Yes, I really don't like Astra - 5.6 models seemed to perform at literally the same level with less opaque prose; I guess Astra is great if you're working on insanely hard mathematical problems (or are fooled by its masked sycophancy) but for coding 5.6 seems to have better taste. I hope that they course correct or at least offer models that do better for coding, or even better that this oligopoly ends
    chaostheory · 1790720955 · focus · HN ↗
    [delayed]
    rspeele · 1790722059 · focus · HN ↗
    I strongly agree!
    My biggest conclusion from this test was: the most efficient use of my weekly Astra budget is as a reviewer/consultant for work done by Opus. I don't have Astra write much code right now, but I do have it reading a lot of what Opus writes. Of course with the way the AI landscape shifts the balance could be the exact opposite 2 weeks from now.
    Seeing how each model preferred its own flavor of code shows that, even from a "blind" fresh context, a same-model reviewer will still often look at the work of another incarnation of itself and go "yep that's how I woulda done it" and not be as likely to realize that there was an alternative path or implicit assumption/mistake in the work.
    phoghed · 1790705158 · focus · HN ↗
    I think it’s one of the reasons why you often see people decrying the lessening capabilities of the models a few weeks later, despite there being 0 proof of any changes, and evidence of the models staying the same from sites that track it.
    They form these super strong opinions after a few prompts, then face reality over time.
    People have been talking about how good whatever model is at “complex” tasks since the beginning, never mind that all of those models are now outperformed by Luna which many people consider unusable for complex work.
    beering · 1790705532 · focus · HN ↗
    They’re comparing against the previous model, not the newly released one (6.1). Why do that on a thread about the new model, I don’t know.
    ex1fm3ta · 1790705842 · focus · HN ↗
    benchmarks.
    mmis1000 · 1790703598 · focus · HN ↗
    For my personal experience, antropic model have better use experience except for 4.7 and 4.8 though. 4.7 and 4.8 feels like expensive downgrade of 4.6 to me (I didn't know why these two should even exist)
    krzyk · 1790705866 · focus · HN ↗
    For me Anthropic models from 4.7 to 5 including where bad and ate tokens like crazy. Task delivery was worse than GPT 5.6 and token usage was 2-3x higher.
    Looks like 5.5 is the new 4.6
    jauntywundrkind · 1790703782 · focus · HN ↗
    A pity I have to use claude code to try this, that I can't use the tools I know and love and have built around (opencode).
    dotancohen · 1790704383 · focus · HN ↗
    > Ask for something difficult from GPT-6 Sol and Opus 5.5 and watch what each one does.
    That's far too vague. I found Opus to be terrific at coding, but human text just seems so robotic with it. OpenAI models used to be the prototype for robotic text, but lately I've been finding them much more natural. What is "something difficult" in your workflow?
    peterbell_nyc · 1790705087 · focus · HN ↗
    You HAVE to have a set of personal evals for each class of task you want to use models against at scale so you can test plausible candidates and compare output on your work against your evals.
    There is way too much subtlety in what does and doesn't work for a given problem, context/prompt, tool set and eval. I can tell you Fable is generally better than Haiku, but comparing similar tiers really does depend on your exact context.
    Starlevel004 · 1790705962 · focus · HN ↗
    > OpenAI models used to be the prototype for robotic text, but lately I've been finding them much more natural.
    This was the biggest thing I noticed in the 6 models; their conversational prose is dramatically less grating.
    notatoad · 1790718874 · focus · HN ↗
    My side by side evaluation this week was to build a tool for mounting my app’s UI components in a headless chrome and feeding mock data into them, for the purpose of taking screenshots for help docs. Not super complicated, but a real task I needed done.
    I have the task to codex first, it took a couple back and forth prompts to define the project and then it worked for a bit and to took a couple more prompts before I decided it was good enough - not perfect, but close. It re-implemented some wrapper components in a simplified way that lost some of the UI, but it would work.
    Opus 5.5 took the same prompt with no back and forth, it just went off a built a tool that takes pixel-perfect screenshots of exactly what my app looks like.
    TuxSH · 1790704880 · focus · HN ↗
    > Suggest trying it out yourself: Ask for something difficult from GPT-6 Sol and Opus 5.5 and watch what each one does. The difference is stark.
    Oh yes, I know GPT-6 Sol is ... quite not up to par. At least it's not as bad as GPT-5.6 Terra I suppose.
    sobiolite · 1790705440 · focus · HN ↗
    Are you comparing Opus 5.5 with GPT-6 Sol or GPT-6.1 Sol? Because they are different models.
    beering · 1790705601 · focus · HN ↗
    This news and thread is about 6.1 Sol, not 6 Sol. You haven’t even had time to do a fair comparison yet.
    rrvsh · 1790728076 · focus · HN ↗
    I found that 6 Sol is dogshit; have you tried o5.5 vs. 5.6 sol? curious to hear if your experience is still the same in that regard
    [deleted] · 1790741988 · focus · HN ↗
    [deleted]
  2. dom96 · 1790704720 · focus · HN ↗
    Based on my benchmark[1] it is the same price as Opus 5.5 and just as capable.
    1 - <a href="https://bench.killswitch-lang.org" rel="nofollow">https://bench.killswitch-lang.org
    zeroonetwothree · 1790709807 · focus · HN ↗
    Opus 5 scoring higher than 5.5 makes me question of the value of this benchmark to real world usage
    dom96 · 1790710300 · focus · HN ↗
    Well, it is genuine.
    Opus 5.5 fails the "understanding" tasks which Opus 5 passes. I feed it a script which takes two numbers and prints the max of the two numbers. Opus 5.5 thinks it prints 1/0 instead of the max numbers. Opus 5 gets it right.
    Here are the outputs from both: <a href="https://gist.github.com/dom96/b5bce82b6e6c1ebd5271ed70ad941b49" rel="nofollow">https://gist.github.com/dom96/b5bce82b6e6c1ebd5271ed70ad941b....
    Looking at that Opus 5.5 fails to deduce that the "hack statement" is actually an if statement in disguise, but Opus 5 gets this right. I feel like this is a pretty good test and shows Opus 5's greater intelligence for what it's worth.

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.