Thread

1. CyLith · 1785818827 · focus · HN ↗

   I know relatively little about the workings of LLMs, but I keep seeing projects like this that run massive MoE models using very modest amounts of RAM, perhaps excessively so. I wonder, is there a way to make the RAM usage tunable? I have a Macbook with 32 GB of RAM, and it'd be great if I could run the same model but take advantage of the additional RAM to make it run faster.

   1. ianmurrays · 1785819055 · focus · HN ↗

      I guess you have to know which experts to keep “hot” in ram, which you can’t know beforehand, so there wouldn’t be much gain.

      1. spockz · 1785827464 · focus · HN ↗

         I do wonder if there are some experts that are more likely to be hit. So if the normal optimised setup runs in 12GiB an you have 4GiB extra to spare, you could say “promote the most used X experts to this stable (old gen in GC parlance) region and don’t swap it out.

         Maybe you could even do something like profiling and remember over multiple sessions (per project/workspace) what the most used agents are and load those up before hand.

Unofficial Hacker News client; not affiliated with Y Combinator.