Thread

1. herf · 1790792835 · focus · HN ↗

   I have two NVIDIA GPUs (16GB+16GB) here, and it detects them each twice (says I have 4 GPUs). But then, it says most models are too big (anything >8GB?) and seems to run only on one GPU (5070ti).

   Unfortunately even with my 5070ti, llama.cpp seems to be about 20-30% faster at decode, running as:

   set CUDA_VISIBLE_DEVICES=0 build\bin\Release\llama-server -hf google/gemma-4-12B-it-qat-q4_0-gguf -ngl 99 --no-mmproj-offload -mg 0 -c 262144 -fa on --host 0.0.0.0

   1. sheephess44 · 1790824599 · focus · HN ↗

      [dead]

Unofficial Hacker News client; not affiliated with Y Combinator.