Thread

1. entrope · 1789647490 · focus · HN ↗

   A lot of the article focuses on problems induced by a 512-token input limit. For example, one needs a lot more chunks with such a small input, especially with overlap. I realize that some embedding models do have input contexts that small, but 8K and 32K are fairly widely supported and reduce chunking-related problems.

   For languages like English, there's also usually a lot of redundancy within a text, so 512 tokens might not give a very clear indication of the context. Lots of documents have similar introductions (like "#include <foo.h>\n") that make short contexts and truncation particularly harmful.

   Also, "Nothing in the document past that point can ever be retrieved, and nothing anywhere told you." This is user-hostile behavior, even if they didn't want to admit to users that the auto-embedding support was poor.

   Finally, the paragraph later on about truncation being "what you already have" reads like Claude talking to the developer, not like a vendor talking to users. But sure, maybe this is a good default for a database searching page titles, chat logs and Xeets?

   1. PaulHoule · 1789659029 · focus · HN ↗

      "Chunking" has a bad history in IR going back through the history of the TREC conference. For instance you might index the whole document or index sections or paragraphs and somehow aggregate the paragraph hits up into document hits. It's one of the many things that doesn't help performance even if you think it did, and it may hurt.

      The answer is "make the context window as large as you reasonably can" because you have to have enough (con)text in the window for the system to decide what the words mean and if you don't have enough of it you won't get it right.

Unofficial Hacker News client; not affiliated with Y Combinator.