‹ BackHN Continuity

Thread

From the creator of Redis; run LLM locally with ds4

359 points · 103 comments · fibo

  1. TheqO · · focus · HN ↗

        Metal, the primary target, on Macs with 96 GB or more. Smaller machines can use SSD streaming. SSD streaming is also needed in order to run very large models such as full GLM 5.x (not Flash) on 128GB systems
    Anyone tested token speeds at less than 96gb RAM on apple?
Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.