Thread

Context Language Models

176 points · 51 comments · emersonmacro

_jayhack_ · 1790913552 · focus · HN ↗
Letting a model manage its own context is very bitter lesson-pilled
Biggest challenge is you will get a much lower cache hit rate if you frequently edit the agent's context/prefix, so this can not be implemented efficiently via e.g. the Anthropic API.
This ^ can be solved in principle but likely requires modifications to the transformer architecture and definitely to serving infrastructure
See related: "KV Cache Rules Everything Around Me": <a href="https://www.completeskeptic.com/p/kv-cache-rules-everything-around" rel="nofollow">https://www.completeskeptic.com/p/kv-cache-rules-everything-...
1. wangii · 1790926421 · focus · HN ↗
  is kvcache a prematural optimization?
  1. f_devd · 1790926941 · focus · HN ↗
    It's pretty much required to make it work with large contexts, so probably not
  2. gfrecvh · 1790937203 · focus · HN ↗
    Do you even understand what you're suggesting? Do you understand how the data flow would look like without a kv cache?
    wangii · 1790941149 · focus · HN ↗
    Java spent decades making pointer-rich identity semantics cheaper, before increasingly recognizing that many things programmers call “objects” are actually values.
    We may spend a decade making persistent token-level KV state cheaper, before recognizing that much of what we call “context” should actually be transient, compressed, reconstructed, or represented in an entirely different state space.
    KV cache risks turning an optimization of today's representation of history into tomorrow's semantics of memory.
  3. wat10000 · 1790947342 · focus · HN ↗
    Not having it would be like a text editor that loads your entire document from disk and re-saves it every time you press a key. On a floppy drive.
    wangii · 1790990853 · focus · HN ↗
    [dead]

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.