Thread

1. vatsachak · 1790797844 · focus · HN ↗

   I wonder what the "waves" in LLMs are; The global firing patterns that can help us interpret what the weights are doing en-masse

   1. randomImmigrant · 1790809346 · focus · HN ↗

      Why would you assume waves with LLMs? What about the structure or function of an artificial neuron prompts you to hypothesize they generate continuous waves in a bunch of nested frequency ranges?

      1. peddling-brink · 1790827106 · focus · HN ↗

         > "waves"

         OP was postulating that there might be higher order patterns we can see arise out of underlying LLM mechanics.

         1. randomImmigrant · 1790828269 · focus · HN ↗

            But there is no LLM mechanics. You can certainly take measurements along the neural net and show a wave like pattern of activation. But no such wave can be measured in the hardware. An artificial neuron is really doing nothing to produce physical weave behavior that’s measurable.

            1. jibalt · 1790851465 · focus · HN ↗

               Just as the "neurons" are virtual, the "waves" would be virtual. They aren't talking about physical waves produced by the hardware. Likewise the LLM "mechanics" are virtual--it's nonsensical to say they don't exist.

               1. randomImmigrant · 1790865790 · focus · HN ↗

                  Yeah no. “Virtualizing” isn’t some math function that will automatically preserve dynamics after it is applied.

Unofficial Hacker News client; not affiliated with Y Combinator.