‹ BackHN Continuity

Thread

From the creator of Redis; run LLM locally with ds4

359 points · 103 comments · fibo

  1. vlowther · · focus · HN ↗
    It is pretty nifty. I spend some time over last weekend implementing fused TQ to allow for 1m context lengths on a 128 gb MacBook M5 Max when using Qwen 3.8 flash next (<a href="https:&#x2F;&#x2F;github.com&#x2F;antirez&#x2F;ds4&#x2F;pull&#x2F;1115" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;antirez&#x2F;ds4&#x2F;pull&#x2F;1115 if you are interested). If I get bored I might port over the Metal kernels from oMLX -- the speed increase they have for the v0.7.0 release is amazeballs.
    1. ttoinou · · focus · HN ↗
      I’m already able to use 1M context windows with the same machine than you and same model. Strange
      1. vlowther · · focus · HN ↗
        Yeah, most of what I did was to add fused TQ support to leave more memory free for other nefarious purposes.
        1. ttoinou · · focus · HN ↗
          Oh that’d be nice indeed
Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.