Thread

Show HN: Janus – Go binary that runs GGUF models via Vulkan on AMD/Intel/Nvidia

104 points · 19 comments · Maverick617

Loading the complete thread in the background. This saved snapshot is available now. Refresh

PcChip · 1790889556 · focus · HN ↗
I didn't see any benchmarks against vllm, sglang, exllama, etc
1. rancor · 1790890078 · focus · HN ↗
  Since this is basically a wrapper around libllama.so, I would assume that the performance is roughly the same as llama.cpp upstream.
  1. nullpoint420 · 1790914499 · focus · HN ↗
    Woof. Wonder if the creator knows that
peddling-brink · 1790894332 · focus · HN ↗
> llama.cpp via Vulkan (AMD / Intel / NVIDIA) or CPU fallback
I got excited about someone paying attention to intel. Oh well.
1. kamranjon · 1790899736 · focus · HN ↗
  llama.cpp sycl and vllm xmx work is pretty incredible right now - you just gotta build it with some extra flags
  1. peddling-brink · 1790907985 · focus · HN ↗
    Llama would be nice for the ggufs. Any specific flags or tutorials I should look at?
    gunalx · 1790924445 · focus · HN ↗
    The docs are a great start.
    <a href="https://github.com/ggml-org/llama.cpp/blob/master/docs/backend/SYCL.md" rel="nofollow">https://github.com/ggml-org/llama.cpp/blob/master/docs/backe...
2. wronglebowski · 1790904870 · focus · HN ↗
  What hardware do you have? I’ve been playing with a 258V and OpenVINO has come a longggggg way.
  1. peddling-brink · 1790907870 · focus · HN ↗
    Two arc b60s. The intel vllm build is getting me ~15t/s decode with heavy context using qwen3.8 27b.
    bitexploder · 1790951219 · focus · HN ↗
    Those cards should be able to do a lot more. They support int8 and they have very good processing speed. For reference I have a single V100 running around 900t/s prefill and 100 t/s decode during DeepSWE runs on 27B (4 bit quant). Those cards have half the bandwidth so a realistic decode is going to stop around 50 t/s, but those cards can support NVFP4 more easily than the V100. Combined with a TP build you should be able to get 100 t/s. And you should be able to 2x the prefill of a V100. I have had Claude optimizing llama.cpp for about a week and am at around 300 t/s decode so far on two ancient V100s @ 64 GB RAM.
    Also for the 3 people that ever read this and are curious about local models still, Qwen 27B 3.8 matched Sonnet 5 in the 17 DeepSWE tasks I have run so far, solving the exact same 7 it has. Caveat: datacurve combined low/medium/high Sonnet 5 data.
    wingtw · 1791012828 · focus · HN ↗
    Very interesting! Will you be sharing the optimizations ?
    bitexploder · 1791048462 · focus · HN ↗
    I will make sure they get out there soon :)
dlcarrier · 1790894409 · focus · HN ↗
From what I've seen, Vulkan adds a lot of overhead on Intel hardware.
1. gunalx · 1790924172 · focus · HN ↗
  Yes and no. I tested llamacpp on my intel gpu with both vulkan and sycl. I measured them to be in the same ballpark even if i have se en pepole claim marger differences than i observed.
  In the end i took the sligth slowdown of vulkan to have a more stable and higher development velocity backend While being able to use identical setups on both intel and and gpus.
shayanjavadi · 1790897305 · focus · HN ↗
[dead]
aidiveyt · 1790922544 · focus · HN ↗
claude code appends a role:"system" block after the user prompt, so a proxy rewriting the trailing user message is a no-op.
DylanMerigaud · 1790924844 · focus · HN ↗
Intel hardware can have Vulkan overhead, impacting performance.
antonyragleap · 1790933338 · focus · HN ↗
Curious about Vulkan overhead on Intel vs AMD/Nvidia for long context. Any benchmarks vs vllm/sglang?
ContinuityLab · 1790942614 · focus · HN ↗
[dead]
kgeist · 1790957016 · focus · HN ↗
This seems to be a wrapper around libllama, what's the point? Llama.cpp already ships a web server. I don't see anything in the README that llama.cpp doesn't already support.
Usually people advertise "Go binary" when it's pure Go, not just a CGO wrapper. The project seems to be the result of 5 minutes of running Claude Code.
colinsane · 1790957897 · focus · HN ↗
> Local inference — llama.cpp via Vulkan (AMD / Intel / NVIDIA) or CPU fallback
so this is just a go binary that exec's `llama-server --model $MODEL_PATH ...`? there's room for llama wrappers, sure, but the readme only shows features that are already exposed directly from llama-server. it doesn't seem to do anything besides rename the CLI arguments: i don't get it.

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.