Thread

1. kamranjon · 1789680375 · focus · HN ↗

   Love this for the folks with 16gb graphics cards - 3.8 27b has been incredible but not quite runnable on anything less than 32gb - will try loading this up on my 16gb intel b50 and see how it goes - not sure these quants can be accelerated by the XPU cores yet but maybe in time!

   1. kadoban · 1789681996 · focus · HN ↗

      You can run the ~4 bit quant(s) on 24gb, if you're not _too_ picky on context size.

      This will hopefully be better, though it'd be a _very_ surprising increase in performace at the size they say. Would love to see more about how it benchmarks.

      1. redox99 · 1789699167 · focus · HN ↗

         You can trivially run 131k on 24GB 4bit, and there are repos with tweaks that allow you to get the full 262k but idk if there's degradation with their approach.

         1. Zambyte · 1789734799 · focus · HN ↗

            How? I'm running 4bit with a q8 kv on a 24gb card, and I'm not able to get 100k out of it. I use a context size of 90k.

            1. [deleted] · 1790012305 · focus · HN ↗

               [deleted]

Unofficial Hacker News client; not affiliated with Y Combinator.