Thread

1. drtgh · 1789758284 · focus · HN ↗

   > relatively poorly understood technology

   Poorly understood? how convenient...

   LLMs are vectorial databases with losses that index statistically filled data, which uses a text interface to query such statistically filled data. The output is a string concatenation (statistically concatenated bit by bit).

   When the LLMs are queried (prompted), you can get random mixed data as output, ERRORS, due to undesired indexes getting closer at one point while the string was being concatenated for the output, what affects the rest of the indexed content that will be concatenated.

   It is intrinsic to this tech. The larger the context, the greater the probability of get mixed data. And if the provider lowers the precision of those indexes -in order to decrease hardware resources and energy consumption- such probability increases to the point where those errors are granted.

   Even knowing that the queries can return wrong/mixed data in the responses, errors, the companies developing this, decided to introduce a new product, that connects such LLMs outputs to the command console, latter connected to internet, raw 'eval' running commands from such outputs witch obviously can contain whatever mixed random. Then we started to hear "oh, it deleted my directory", etc, and it seems the next one will be "a missile killed my wife", because it is a text concatenation engine with errors.

   To name it "hallucination" is an euphemism... those are errors, and they are granted to happen at one moment. If they do not know this, then they ate too much marketing without doing their job, or it was a convenient contract for the pocket$ of someone.

   1. theptip · 1789760419 · focus · HN ↗

      > LLMs are vectorial databases

      You use a bunch of technical-sounding words here to make it sound like you understand. But to be clear, nobody understands why the evolved weights of a NN make the decisions that they do.

      Almost nothing is understood about the actual representations used for nontrivial concepts, decision algorithms, etc.

      If you look at the field of mechanistic interpretability, compared to “GOFAI” like learned decision trees, an LLM is completely opaque.

      1. coldtea · 1789768236 · focus · HN ↗

         >But to be clear, nobody understands why the evolved weights of a NN make the decisions that they do.

         We might not understand particular "emergent" capabilities, but the low level mechanism is not just understood, but a deterministic algorithm with a handful of basic componets, that are well understood themselves.

         1. conscion · 1789768674 · focus · HN ↗

            > We might not understand particular "emergent" capabilities

            The emergent capabilities are the only capabilities we care about

            1. coldtea · 1789768930 · focus · HN ↗

               For allignment maybe.

               For the core functionality and the optimizations we don't really need to know how the emergent capabilities decide on particular answers.

               Which is why we could build LLMs before those features ...emerged for us to see, and why we can just code LLMs with the numerical NN algorithms we use, and do now have to go in and change individual weights.

               1. semiquaver · 1789784290 · focus · HN ↗

                  Paraphrasing

                      Me: it’s disturbing we don’t know why this pile of numbers we made seems to *think* in a way previously only done by humans. I think it’s important that we understand this better if possible. 
                  
                      You: we don’t really need to know why that happens.
                  

                  We don’t? I sure would like to know!

                  1. bigyabai · 1789842238 · focus · HN ↗

                     We do understand "thinking" though. That's literally the whole point of Attention is All You Need, the attention mechanism is what separates the transformer architecture from other neural networks. It's well worth a read if you haven't gone over it yet.

                     Features like chain-of-thought, long-horizon contexts and RoPE/YaRN all extend this thinking capability very transparently. The only remaining thing to study is the data and weights, which probably isn't going to contain some sort of miraculous revelation.

                     1. theptip · 1789867466 · focus · HN ↗

                        Attention is at about the same level of abstraction as spike trains or action potential, IMO. It’s a mechanism, it doesn’t tell you much at all about how actual concepts get represented. (It merely defines the substrate with which they can be represented.)

                        The entire field of Mechanistic Interpretability exists because just understanding Attention does not in any way help you to understand why a certain NN responds with a certain hallucination about a certain Chinese boat in this specific context.

                        > The only remaining thing to study is the data and weights

                        To me this is like saying “the only thing left to study in the brain is the connectome; probably going to be boring, we understand it already”. It’s almost all of the hard/meaningful stuff! It’s where intelligence and consciousness lives!

                        1. bigyabai · 1789870686 · focus · HN ↗

                           The entire field of mechanistic interpretability has gotten nowhere. The formalized, "causal" understanding of latent space conceptualization isn't any better than an LLM cargo cult.

                           It's wholly possible that you could study one set of weights for decades, and find nothing. There's no guarantee that any patterns outside of human language exist in that data. In this specific context, it's satisfying enough to state that [Chinese] and [boat] were both tokens in the tokenizer, activated by a feedforward pass through weights that favor [boat] after [Chinese]. There's not any guaranteed solution to this. There's not even any guaranteed problem; that hallucination is an expected behavior.

Unofficial Hacker News client; not affiliated with Y Combinator.