Thread

1. peddling-brink · 1790894332 · focus · HN ↗

   > llama.cpp via Vulkan (AMD / Intel / NVIDIA) or CPU fallback

   I got excited about someone paying attention to intel. Oh well.

   1. wronglebowski · 1790904870 · focus · HN ↗

      What hardware do you have? I’ve been playing with a 258V and OpenVINO has come a longggggg way.

      1. peddling-brink · 1790907870 · focus · HN ↗

         Two arc b60s. The intel vllm build is getting me ~15t/s decode with heavy context using qwen3.8 27b.

         1. bitexploder · 1790951219 · focus · HN ↗

            Those cards should be able to do a lot more. They support int8 and they have very good processing speed. For reference I have a single V100 running around 900t/s prefill and 100 t/s decode during DeepSWE runs on 27B (4 bit quant). Those cards have half the bandwidth so a realistic decode is going to stop around 50 t/s, but those cards can support NVFP4 more easily than the V100. Combined with a TP build you should be able to get 100 t/s. And you should be able to 2x the prefill of a V100. I have had Claude optimizing llama.cpp for about a week and am at around 300 t/s decode so far on two ancient V100s @ 64 GB RAM.

            Also for the 3 people that ever read this and are curious about local models still, Qwen 27B 3.8 matched Sonnet 5 in the 17 DeepSWE tasks I have run so far, solving the exact same 7 it has. Caveat: datacurve combined low/medium/high Sonnet 5 data.

            1. wingtw · 1791012828 · focus · HN ↗

               Very interesting! Will you be sharing the optimizations ?

               1. bitexploder · 1791048462 · focus · HN ↗

                  I will make sure they get out there soon :)

Unofficial Hacker News client; not affiliated with Y Combinator.