Thread

1. lxe · 1791128170 · focus · HN ↗

   Why isn't this type of expert caching in the native llama.cpp yet? Why do we need a separate codebase?

   1. hgoel · 1791129281 · focus · HN ↗

      The "mainstream" inference engines are notoriously slow to integrate this stuff, to an extent understandably given the complexity of ensuring numerical accuracy alongside supporting a wide array of systems and models. Part of it is that not everyone is willing to bring what they develop into a pull request because they vibe coded it and don't care to deal with whatever quality requirements the more well known inference engines have.

Unofficial Hacker News client; not affiliated with Y Combinator.