Thread

1. vatsachak · 1790797844 · focus · HN ↗

   I wonder what the "waves" in LLMs are; The global firing patterns that can help us interpret what the weights are doing en-masse

   1. randomImmigrant · 1790809346 · focus · HN ↗

      Why would you assume waves with LLMs? What about the structure or function of an artificial neuron prompts you to hypothesize they generate continuous waves in a bunch of nested frequency ranges?

      1. peddling-brink · 1790827106 · focus · HN ↗

         > "waves"

         OP was postulating that there might be higher order patterns we can see arise out of underlying LLM mechanics.

         1. randomImmigrant · 1790828269 · focus · HN ↗

            But there is no LLM mechanics. You can certainly take measurements along the neural net and show a wave like pattern of activation. But no such wave can be measured in the hardware. An artificial neuron is really doing nothing to produce physical weave behavior that’s measurable.

            1. jibalt · 1790851465 · focus · HN ↗

               Just as the "neurons" are virtual, the "waves" would be virtual. They aren't talking about physical waves produced by the hardware. Likewise the LLM "mechanics" are virtual--it's nonsensical to say they don't exist.

               1. randomImmigrant · 1790865790 · focus · HN ↗

                  Yeah no. “Virtualizing” isn’t some math function that will automatically preserve dynamics after it is applied.
         2. HarHarVeryFunny · 1790855921 · focus · HN ↗

            Brain waves are due to feedback paths and dynamics - oscillations etc, but LLM connectivity is far simpler. In an LLM there is a strict bottom-up one-way flow of information, with no looping. There is no way for a higher up layer to affect a lower down layer, whether directly or indirectly.

Unofficial Hacker News client; not affiliated with Y Combinator.