I've been running Qwen 3.8 27b (an opus 4.6 tier model), locally on a 5090 for just over two weeks @ 170 tokens/s. That's a frontier model from 9 months ago running on consumer hardware. Who knows where distillation and pruning gets us in another year.
Weird, I kinda gave up on 3.8 as anything other than a planner. I had it try to write some basic unit tests for an admittedly complex bit of code and it ran out of context thinking about the problem and exploring random parts of the code base repeatedly before it even wrote a single line. Toning down the thinking helped some, but then it wasn't much better than qwen coder.
Have you attempted some of the "swift" variants of 3.8? I've heard they're super good in terms of toning down thinking without affecting performance
The only swift variant I'm aware of is from ukisai, which uses the swift open license, and is NOT open for commercial use for businesses over $1mil. I'm respecting their choice by not using it for work, which means I'm also not using it for my personal projects in case I forget to switch models when I switch projects.
My day job doesn't have a dedicated enterprise contract with any of the ai vendors so I might trial it to see if it is worth promoting at the company. Part of me is still holding out hope that qwen 4 dials back the overthinking on its own.
> The only swift variant I'm aware of is from ukisai, which uses the swift open license,
Ah, I overlooked that, since I have a claude sub at work and all my explorations are purely personal. There's another fast version of 3.8: ThinkingCap[1] by bottlecapai but they have the same $1M restriction from what I can see since it's distributed under a PolyForm Small Business 1.0.0 + BottleCap personal-use grant.
reedf1 · · focus · HN ↗
zdragnar · · focus · HN ↗
the_lucifer · · focus · HN ↗
zdragnar · · focus · HN ↗
My day job doesn't have a dedicated enterprise contract with any of the ai vendors so I might trial it to see if it is worth promoting at the company. Part of me is still holding out hope that qwen 4 dials back the overthinking on its own.
the_lucifer · · focus · HN ↗
Ah, I overlooked that, since I have a claude sub at work and all my explorations are purely personal. There's another fast version of 3.8: ThinkingCap[1] by bottlecapai but they have the same $1M restriction from what I can see since it's distributed under a PolyForm Small Business 1.0.0 + BottleCap personal-use grant.
[1]: <a href="https://huggingface.co/bottlecapai/ThinkingCap-Qwen3.8-27B" rel="nofollow">https://huggingface.co/bottlecapai/ThinkingCap-Qwen3.8-27B