I’m on a strix halo @ GPU-5 with MTP and I get 600 prefill and 30 TG which pushes it into a very usable range. The odd thing is that Dflash2 is really slow for me, like sub 10 TG.
I've found the opposite on my R9700 (n-max=7, no other speculative decoding like ngram-mod, which I found slows it down). I think it depends whether your workload and system are bandwidth limited or compute limited. I see draft acceptance around 0.55, so 0.55 * 7 = 3.8 tokens per pass, which on my bandwidth-limited card takes me from 30tps to a peak of 80tps on llama.cpp (MTP peaked at ~65tps). I'm also running a Qwen fine tune whose speculative execution is better than the base model.
Strix Halo has lower compute than the R9700 but the RAM is also slower, so not sure what would be the ultimate limiting factor.
I run llama.cpp with -ctv=8, -ctk=q4. Vulkan has better throughput if you're doing single-stream decode, but ROCm has better throughput if you have "--parallel 2" or higher. If supporting parallelism, unified kv cache should be off, especially with Vulkan.
Of course, some of these may be specific to my card so try variations for your hardware. Hermes can concoct a test suite and run some tests for different llama.cpp parameter permutations to find something optimal.
syntaxing · · focus · HN ↗
naasking · · focus · HN ↗
Strix Halo has lower compute than the R9700 but the RAM is also slower, so not sure what would be the ultimate limiting factor.
syntaxing · · focus · HN ↗
naasking · · focus · HN ↗
DFlash2, Q8_0, --spec-draft-n-max=7: <a href="https://huggingface.co/z-lab/Qwen3.8-27B-DFlash2-GGUF" rel="nofollow">https://huggingface.co/z-lab/Qwen3.8-27B-DFlash2-GGUF
I run llama.cpp with -ctv=8, -ctk=q4. Vulkan has better throughput if you're doing single-stream decode, but ROCm has better throughput if you have "--parallel 2" or higher. If supporting parallelism, unified kv cache should be off, especially with Vulkan.
Of course, some of these may be specific to my card so try variations for your hardware. Hermes can concoct a test suite and run some tests for different llama.cpp parameter permutations to find something optimal.
syntaxing · · focus · HN ↗