I've been running Qwen 3.8 27b (an opus 4.6 tier model), locally on a 5090 for just over two weeks @ 170 tokens/s. That's a frontier model from 9 months ago running on consumer hardware. Who knows where distillation and pruning gets us in another year.
reedf1 · · focus · HN ↗
redanddead · · focus · HN ↗
off_with_their_ · · focus · HN ↗
[dead]