Thread

Qwen3.8 Max now ranked as the best overall model by agentic index

174 points · 72 comments · apitman

Loading the complete thread in the background. This saved snapshot is available now. Refresh

embedding-shape · 1786043690 · focus · HN ↗
Strange that the page <a href="https://artificialanalysis.ai/agents/coding-agents" rel="nofollow">https://artificialanalysis.ai/agents/coding-agents doesn't even mention "Qwen" once if it's now the "best" according to one of their one index?
1. scrlk · 1786043941 · focus · HN ↗
  Different benchmarks:
  > Artificial Analysis Agentic Index: Represents the weighted average of agentic capabilities benchmarks in the Artificial Analysis Intelligence Index (GDPval-AA v2, Tau³-Banking)
  > Artificial Analysis Coding Agent Index v1.3 incorporates 3 benchmarks: DeepSWE, Terminal-Bench v2, and SWE-Atlas-QnA
  1. apitman · 1786044107 · focus · HN ↗
    Looks like coding agent is model+harness. There are far fewer models represented on that page. I believe "agentic index" is still the metric to look at for coding performance. I could be wrong about that though.
2. Bootvis · 1786044058 · focus · HN ↗
  Indeed, and this Qwen 3.8 max specific page:
  <a href="https://artificialanalysis.ai/models/qwen3-8-max" rel="nofollow">https://artificialanalysis.ai/models/qwen3-8-max
  Doesn't have the claim either. Clickbait?
  1. petu · 1786044252 · focus · HN ↗
    This page has it, scroll to "Intelligence" header and click "Agentic Index"
    Bootvis · 1786044643 · focus · HN ↗
    So the original link should be: <a href="https://artificialanalysis.ai/models/qwen3-8-max?intelligence=agentic-index" rel="nofollow">https://artificialanalysis.ai/models/qwen3-8-max?intelligenc...
    Even then, this seems a much more marginal win than the headline suggested to me.
3. amelius · 1786044078 · focus · HN ↗
  According to those graphs, Grok 4.5 appears to be the most cost-effective model.
4. artemisart · 1786044350 · focus · HN ↗
  They didn't run all benchmarks. It's the best in AA agentic index (GDPval-AA v2, ³-Banking) but not coding index (DeepSWE which is missing, Terminal-Bench v2.1 they have 81% vs 90% for Sol, SWE-Atlas-QnA missing).
5. moritzwarhier · 1786044736 · focus · HN ↗
  [delayed]
syntaxing · 1786043706 · focus · HN ↗
I am so excited for Qwen 3.8 27B. It’s a shame how slow prefill (~3-400) is on a strix halo but it’s such a good model for agentic tasks.
1. tarr11 · 1786044196 · focus · HN ↗
  What type of agentic tasks are you using it for (eg how complex)?
2. CamperBob2 · 1786044360 · focus · HN ↗
  How are you running it on a Strix Halo? The weights aren't out yet, are they?
  1. 13rac1 · 1786044572 · focus · HN ↗
    I interpret @syntaxing as meaning they are looking forward to running Qwen3.8-27B, but are frustrated by prefill times with other models, such as Qwen3.6-27B.
  2. syntaxing · 1786044703 · focus · HN ↗
    I meant Qwen3.6. Unsloth supposedly has early preview of the model and the VRAM requirement is the same so most people expect similar model size and type.
3. LoganDark · 1786045583 · focus · HN ↗
  I find that 35B-A3B is much easier to run on my M4 Max (both prefill and generation)
  1. markasoftware · 1786046091 · focus · HN ↗
    It's well known 35b is much faster (on any hardware) and quite a bit dumber
4. colingauvin · 1786046414 · focus · HN ↗
  Prefill is survivable if you cache well. But what kills me is the context. Qwen 27 needs a ton of room for KV Cache. I guess not an issue on a 128 GB Halo or Spark, but if you are running of consumer/prosumer GPUs it's miserable to be compacting every 120k tokens.
drnick1 · 1786043769 · focus · HN ↗
Why does an open weights model cost nearly the same as GPT5.6? $1.14 vs $1.23 on the cost index. Since you can't presumably run this on your own hardware given the model size and hence gain other things like privacy, I don't see any reason to move away from GPT at this rate.
1. eli · 1786043818 · focus · HN ↗
  It's not enough that it's better?
  1. drnick1 · 1786044982 · focus · HN ↗
    > It's not enough that it's better?
    It's barely better, and barely cheaper, not really enough to challenge the status quo IMO.
  2. Art9681 · 1786045784 · focus · HN ↗
    They cherrypicked benchmarks. The ONE weighed benchmark where is beats Opus5 by 0.1 points is what was linked because that's how propaganda works. The Agentic Index that includes the full benchmark suite has it in 5th place.
    Might as well use gpt-sol.
2. apitman · 1786043949 · focus · HN ↗
  For one thing, providers of open models can't arbitrarily increase their prices without facing competition.
  1. frereubu · 1786044050 · focus · HN ↗
    But given the extremely low cost of switching, why wouldn't you use the cheaper one if they're comparable?
    apitman · 1786044248 · focus · HN ↗
    As low as it is, switching between providers on OpenRouter is still lower.
    That said, it's a fair point. For me, it boils down to things covered here: <a href="https://earendil.com/posts/session-portability/" rel="nofollow">https://earendil.com/posts/session-portability/
    Things like obscured reasoning traces.
    copperx · 1786044741 · focus · HN ↗
    [delayed]
3. [deleted] · 1786044040 · focus · HN ↗
  [deleted]
4. Alpha3031 · 1786044042 · focus · HN ↗
  [delayed]
5. efficax · 1786044335 · focus · HN ↗
  it's a big honking trillion some parameters model. it's not cheap to run
6. jjice · 1786045052 · focus · HN ↗
  Qwen Max is their large model - over a trillion params. Similar to Kimi K3 in size. Qwen 3.8 27B is going to be more accessible to your own hardware. I'd say that Qwen Max is not approachable for the majority of people and companies to self-host.
7. criley2 · 1786045650 · focus · HN ↗
  GPT5.6Sol completes the suite in 70M tokens, while Qwen3.8Max needs like 145M tokens. So this is a case where models like Qwen 3.8 and Kimi K3 use a lot more output (reasoning) tokens, go a good bit slower, and then ultimately achieve a better intelligence score than if they went more quickly.
8. ecocentrik · 1786046225 · focus · HN ↗
  Why should open weights correlate with cost? Cost correlates with the expense of running the model more than it does to the expense of developing the model.
9. jazzyjackson · 1786046674 · focus · HN ↗
  Running a large model on rented GPU is still meaningfully more private than handing your chat logs over to FAGA
eli · 1786043776 · focus · HN ↗
I believe it. It's extremely good at troubleshooting. I gave Qwen and Kimi K3 the same annoying, complicated, intermittent bug to track down. Kimi did a bit better in understanding the existing code, but Qwen built some diagnostic tools and did an excellent statistical analysis on the log data. Qwen got way closer to the truth.
I'm very much looking forward to their forthcoming smaller model Qwen 3.8 releases. A version that can easily run locally would be great.
1. comboy · 1786044448 · focus · HN ↗
  How CLI are you guys using for qwen and kimi?
  1. eli · 1786044745 · focus · HN ↗
    I use <a href="https://pi.dev/" rel="nofollow">https://pi.dev/ which works fine out of the box but is fairly minimal and intended to be customized. There are many extensions.
    OpenCode or oh-my-pi might make more sense if you just want a batteries-included agent. You can also make Claude Code work with other models without too much work, but I think that's asking for headaches.
    trey-jones · 1786045258 · focus · HN ↗
    I used claude with GLM and it's easy to set up, just hard to find the documentation. No headaches really, unless you want to use it against multiple different APIs.
  2. g58892881 · 1786046149 · focus · HN ↗
    pi
2. thefourthchime · 1786045525 · focus · HN ↗
  Did you also try Opus 5 and 5.6 Sol?
delduca · 1786043823 · focus · HN ↗
Go China!
SwellJoe · 1786043855 · focus · HN ↗
I find that surprising.
I've been trying it on several projects and have found it's pretty sloppy. It leaves stuff broken, doesn't reliably write tests to check its own work unless explicitly prompted, misunderstands the assignment, etc.
It is smart and reasonably quick but not reliable.
1. dyauspitr · 1786044310 · focus · HN ↗
  It’s because they’re doing some sort of combined score of intelligence, speed and cost. On pure intelligence it doesn’t even show up in the top 10.
2. superfrank · 1786045752 · focus · HN ↗
  I've come to the same conclusion over and over with all of the Chinese models that have been claimed to be catching up with OpenAI's and Anthropic's frontier models (Deepseek 4, GLM 5.2, Kimi K3).
  At their best, I think they're closing in on Opus and GPT, but they're incredibly inconsistent and the variance between their best and their worst is much higher than the best from and Anthropic or OpenAI. The only way I can describe it is that it feels like a lack of intuition with the models which means I find my self needing to write longer prompts or have more back and forth to get them to do what I want from them.
  To give an example, I have a saved prompt that I use as a sanity check on some data I'm storing. It reads about 50 rows from a DB and matches them to the UI and makes sure the data is displaying correctly. I've been using this with GPT 5.5 and now 5.6 for a few months and running it a few times a week with no issue. Sometimes I'll run it multiple times in a single chat if I notice bad data (run it, fix thing, run again, fix another thing).
  I recently tried to switch to using Deepseek v4 (first flash and then pro) and while both did the task just fine, both would do things like change the response format from one message to another in the same chat or randomly decide to omit things it didn't think were relevant. At one point I ran the prompt, fixed some bad data, and then said "Okay, I fixed row 7, run {prompt} again" and so it decided to leave row 7 out of the response. A few times the first message would contain a table and then the next run in the same chat would contain the data in a bulleted list.
  None of those are major issues and all could be solved with a bit more rigor in my prompting, but for me it makes them harder to work with. Those examples are a bit trivial, I think they're the easiest way for me to illustrate the gaps I see with them.
aliljet · 1786043929 · focus · HN ↗
Is there a path to distill this model to do very specific things? Like a RAG strategy for a small (or even large) corpus?
1. Alpha3031 · 1786044350 · focus · HN ↗
  [delayed]
2. teravor · 1786045185 · focus · HN ↗
  once you are able to get the full probability distributions per token you can distill it on specific domains. distilling without that isn't generally a good idea unless you have invested millions in the requisite infrastructure.
brcmthrowaway · 1786044113 · focus · HN ↗
Could someone like Apple be playing the long game - Good Enough(tm) intelligence will eventually fit in our pocket and homes?
1. LPisGood · 1786044897 · focus · HN ↗
  Almost surely. Apple is extremely well positioned to take advantage of this over the next decade.
sirbor · 1786044126 · focus · HN ↗
Qwen is the way to go
dyauspitr · 1786044244 · focus · HN ↗
It doesn’t even show up in the raw intelligence index, so how could it possibly be the best?
quirino · 1786044347 · focus · HN ↗
A couple days ago they had published an overall score of 53 for this model, but that was removed and today it returned with a score of 56.
I wasn't able to find an explanation from them. Anyone knows what happened?
1. Art9681 · 1786045863 · focus · HN ↗
  A wire transfer happened.
  1. ignoramous · 1786046976 · focus · HN ↗
    [delayed]
steve-atx-7600 · 1786044499 · focus · HN ↗
curious about methodology. ive seen them post results for claude/codex when they only ran over benchmarks 3 times per model...
onomojo · 1786044657 · focus · HN ↗
Any benchmark showing Opus 5 as the best just loses credibility for me. Anyone who's actually used Opus 5 daily knows what I'm talking about.
1. copperx · 1786044937 · focus · HN ↗
  I'm dumbfounded to see Opus 5 making SO MANY mistakes in coding simple stuff. Most times, Fable 5 comes out to be cheaper because it nails so many things much quicker than Opus 5.
  1. garciasn · 1786046393 · focus · HN ↗
    I have Fable plan and Opus implement. I haven't had any major issues working this way; however, Opus does seem plain fucking stupid compared to what I experienced with Sonnet previously.
    aenis · 1786046909 · focus · HN ↗
    I do the same, and generally have good results, but it does stupid things with gusto.
    I'd open a blog with "weird things Opus did". Today it launched a swarm of cpu-hogging processes to test if the widget showing machine and I/O load is rendering nicely and correctly. The test went fine, but it was no longer able to kill those processes since they were really effectively hogging the CPU in various ways - being diligent, some of them were hogging CPU, some were murdering the SSD, some were pounding on the network adapters. Took me 30 mins to recover the machine to a working state without killing the meaningful, messy, in-flight sessions i had going on on other projects.
2. visarga · 1786045080 · focus · HN ↗
  Sent to solve one task, came back with half of it solved and 2 more problems.
3. logicchains · 1786045247 · focus · HN ↗
  "As you requested, I've finished task X. Honestly, task X turned out to require task Y, which I haven't actually done. Task Y is the next step if you'd like to continue along this route."
4. cromka · 1786045321 · focus · HN ↗
  Agreed, it's extremely frustrating. It's the only modem that actually makes me curse when talking to it, even knowing how counterproductive it is.
5. bontaq · 1786045680 · focus · HN ↗
  It's an infuriating model
6. nomel · 1786046867 · focus · HN ↗
  What's the clear best, that you see?
7. enraged_camel · 1786046935 · focus · HN ↗
  It's my daily driver. I like it and find it noticeably better than Opus 4.8.
  After I started reading complaints about Opus 5, I gave Fable the task of evaluating a bunch of code Opus 4.8 had written and compare it to Opus 5's code. Fable ran a dynamic workflow and the scores came back 15-20% higher for Opus 5's code in terms of quality, correctness and readability/conciseness. I did not tell Fable which Opus wrote which code, and I turned off memory as well to ensure there was no pollution from that angle.
  My only complaint is that Opus 5's prose is annoying as hell. I wrote a custom skill for it for concise debriefs and it has been working pretty well for me.
petercooper · 1786044913 · focus · HN ↗
Hopefully this boils down to the smaller versions they've teased. In my experience, Qwen models are the closest to the "less knowledge, more intelligence" (yes, the two are hugely correlated!) ideal some tool-dependent tasks need. Even the 3.5 2B can be easily prompted to always lean on tools and not jump to false conclusions (although its actual coding skills are abysmal, as you'd expect).
1. quotemstr · 1786045237 · focus · HN ↗
  > less knowledge, more intelligence
  People produce such models by over-RL-ing smaller models on math and coding tasks. I've found the results capable of neither innovative work nor thinking outside the box. They're straight-A students raised by tiger moments who never let them play freely for hours in the dirt.
  Perhaps you could say such models are skilled --- but intelligent? Not by my measure.
  People and AIs alike need diversity of experience and a broad liberal arts education to see hidden connections between fields and make real advances.
  1. DC-3 · 1786045423 · focus · HN ↗
    It's amusing to me that AI has become sophisticated enough that people have started being racist to it.
looksjjhg · 1786045098 · focus · HN ↗
That took what 2 years? I love how the chip ban made them more efficient
ben8bit · 1786045854 · focus · HN ↗
Haven't tried this yet, but going to soon! I have to wonder what happened at Anthropic. We've cancelled our subscription in favor of OpenCode & Codex. Sol is just so good & OC goes so far for every $ spent. Claude's become a pain to work with - average output with an annoying personality. Who knew this would be an issue even a year ago? In any case, loving the stuff from the Chinese models!
1. tomComb · 1786046174 · focus · HN ↗
  > an annoying personality
  I was with you until there. Qwen and the OpenAI models are great, aggressive agents, but they’re not as good as the anthropic models for human interaction. They just don’t have the subtlety, understanding, or attention to detail.
  1. ben8bit · 1786046419 · focus · HN ↗
    Really? I've heard so many other people complain about this recently. And maybe it's possible that it's the prompt style even. But interesting that it's not across the board.
theropost · 1786045967 · focus · HN ↗
Anthropic is a bit nuts, I had $260 of credits on my max account for the extra usage the other night. It was expiring, so I figured I'll fire up an agentic swarm to deep dive and make some deep changes to some old cold bases.. literally 25 minutes or less, $260 burnt, it didn't get get into the implementation, just wrote a ton of useless plans for the most part. It really opened my eyes to what they expect to charge people.. wayyyy overpriced.
1. polishdude20 · 1786046013 · focus · HN ↗
  You should just spend those towards a cursor subscription.
2. cortesoft · 1786046184 · focus · HN ↗
  It’s crazy how different the credit cost and subscription cost are.
  With the $200 subscription, I can have Fable on ultracode working for hours and not dent the usage limits.
indiantrains · 1786046437 · focus · HN ↗
[dead]
d2p · 1786046613 · focus · HN ↗
I clicked through and it showed Qwen at the top at 55.4 compared to 55.3 for Opus Max. I have a screenshot.
Then I clicked away and back, and now it goes Qwen second, with 58.4, to Opus Max at top with 59.2.
I have screenshots of both. The description above the chart is the same in boh cases:
> Artificial Analysis Agentic Index > Represents the weighted average of agentic capabilities benchmarks in the Artificial Analysis Intelligence Index (GDPval-AA v2, ³-Banking)
What happened? How can the scores change so much in a few seconds?
1. h14h · 1786046732 · focus · HN ↗
  They JUST updated their methodology:
  <a href="https://artificialanalysis.ai/methodology/intelligence-benchmarking" rel="nofollow">https://artificialanalysis.ai/methodology/intelligence-bench...
2. WD-42 · 1786046789 · focus · HN ↗
  Same, they just updated it. Hacker news effect?
atemerev · 1786046730 · focus · HN ↗
Well, that's the bad index then. It is barely usable in my opinion compared to other Chinese frontier models.

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.